You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Shuai Yuan <yu...@supertool.net.cn> on 2010/06/02 04:12:01 UTC
Re: [***SPAM*** ] Re: writing speed test
Thanks lwl.
Then is there anyway of tuning this, faster flush to disk or else?
Cheers,
Kevin
在 2010-06-02三的 09:57 +0800，lwl写道：
> MEM: almost 100% (16GB)
> -----------------
> maybe this is the bottleneck.
> writing concerns Memtable and SSTable in memory.
>
> 在 2010年6月2日 上午9:48，Shuai Yuan <yu...@supertool.net.cn>写
> 道：
> 在 2010-06-01二的 15:00 -0500，Jonathan Shook写道：
> > Also, what are you meaning specifically by 'slow'? Which
> measurements
> > are you looking at. What are your baseline constraints for
> your test
> > system?
> >
>
> Actually, the problem is the utilizaton of resources(for a
> single
> machine):
> CPU: 700% / 1600% (16 cores)
> MEM: almost 100% (16GB)
> Swap: almost 0%
> Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked
> previously)
> NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked
> previously)
>
> So the speed of generating load, about 15M/s as reported
> before seems
> quite slow to me. I assume the system should get at least
> about 50MB/s
> of Disk IO speed.
>
> MEM? I don't think it plays a major role in this writing game.
> What's
> the bottleneck of the system?
>
> P.S
> about Consistency Level, I've tried ONE/DCQUORUM and found ONE
> is about
> 10-15% faster. However that's neither a promising result.
>
> Thanks!
>
> Kevin
>
> >
> > 2010/6/1 史英杰 <sh...@gmail.com>:
> > > Hi, It would be better if we know which Consistency Level
> did you choose,
> > > and what is the schema of test data?
> > >
> > > 在 2010年6月1日 下午4:48，Shuai Yuan
> <yu...@supertool.net.cn>写道：
> > >>
> > >> Hi all,
> > >>
> > >> I'm testing writing speed of cassandra with 4 servers.
> I'm confused by
> > >> the behavior of cassandra.
> > >>
> > >> ---env---
> > >> load-data app written in c++, using libcassandra (w/
> modified batch
> > >> insert)
> > >> 20 writing threads in 2 processes running on 2 servers
> > >>
> > >> ---optimization---
> > >> 1.turn log level to INFO
> > >> 2.JVM has 8G heap
> > >> 3.32 concurrent read & 128 write in storage-conf.xml,
> other cache
> > >> enlarged as well.
> > >>
> > >> ---result---
> > >> 1-monitoring by `date;nodetool -h host ring`
> > >> I add all load together and measure the writing speed by
> > >> (load_difference / time_difference), and I get about
> 15MB/s for the
> > >> whole cluster.
> > >>
> > >> 2-monitoring by `iostat -m 10`
> > >> I can watch the disk_io from the system level and have
> about 10MB/s -
> > >> 65MB/s for a single machine. Very big variance over time.
> > >>
> > >> 3-monitoring by `iptraf -g`
> > >> In this way I watch the communication between servers and
> get about
> > >> 10MB/s for a single machine.
> > >>
> > >> ---opinion---
> > >> So, have you checked the writing speed of cassandra? I
> feel it's quite
> > >> slow currently.
> > >>
> > >> Could anyone confirm this is the normal writing speed of
> cassandra, or
> > >> please provide someway of improving it?
> > >> --
> > >> Kevin Yuan
> > >> www.yuan-shuai.info
> > >>
> > >>
> > >
> > >
> >
>
>
> --
>
> Kevin Yuan
> www.yuan-shuai.info
>
>
>
>
>
--
Shuai Yuan 袁帅
Supertool Corp. 北京学之途网络科技有限公司
www.yuan-shuai.info
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: [***SPAM*** ] Re:
[***SPAM*** ] Re: writing speed test
Posted by Shuai Yuan <yu...@supertool.net.cn>.
Still seems MEM.
However it's hard to convince that constantly writing(even great amount
of data) needs so much MEM(16GB). The process is quite simple,
input_data -> memtable -> flush to disk
right? What does cassandra need so much MEM for?
Thanks!
在 2010-06-02三的 16:24 +0800，lwl写道：
> No.
> But I did some capacity tests about another distributed system.
> Your former test cost too much MEM, it was the bottleneck.
> caches and JVM cost MEM, so I suggested to decrease them.
>
>
> What is the bottleneck of your current test now?
>
>
> 在 2010年6月2日 下午4:13，Shuai Yuan <yu...@supertool.net.cn>写
> 道：
> Hi,
>
> I tried,
>
> 1-consistency level ZERO
>
> 2-JVM heap 4GB
>
> 3-normal Memtable cache
>
> and now I have about 30% improvment.
>
> However I want to know if you have also done w/r benchmark and
> what's
> the result?
>
> 在 2010-06-02三的 11:35 +0800，lwl写道：
>
> > and, why did you set "JVM has 8G heap"?
> > 8g, seems too big.
> >
> > 在 2010年6月2日 上午11:20，lwl <lw...@gmail.com>写道：
> > 3.32 concurrent read & 128 write in
> storage-conf.xml, other
> > cache
> > enlarged as well.
> > --------------------
> >
> >
> > maybe you can try to decrease the size of caches.
> >
> > 在 2010年6月2日 上午11:14，Shuai Yuan
> > <yu...@supertool.net.cn>写道：
> >
> >
> > 在 2010-06-02三的 10:37 +0800，lwl写道：
> > > is all the 4 servers' MEM almost 100%?
> >
> >
> > Yes
> >
> >
> > > 在 2010年6月2日 上午10:12，Shuai Yuan
> > <yu...@supertool.net.cn>写
> > > 道：
> > > Thanks lwl.
> > >
> > > Then is there anyway of tuning
> this, faster
> > flush to disk or
> > > else?
> > >
> > > Cheers,
> > >
> > > Kevin
> > >
> > > 在 2010-06-02三的 09:57 +0800，lwl
> 写道：
> > >
> > > > MEM: almost 100% (16GB)
> > > > -----------------
> > > > maybe this is the bottleneck.
> > > > writing concerns Memtable and
> SSTable in
> > memory.
> > > >
> > > > 在 2010年6月2日 上午9:48，Shuai
> Yuan
> > > <yu...@supertool.net.cn>写
> > > > 道：
> > > > 在 2010-06-01二的 15:00
> -0500，
> > Jonathan Shook写道：
> > > > > Also, what are you
> meaning
> > specifically by 'slow'?
> > > Which
> > > > measurements
> > > > > are you looking at.
> What are
> > your baseline
> > > constraints for
> > > > your test
> > > > > system?
> > > > >
> > > >
> > > > Actually, the problem is
> the
> > utilizaton of
> > > resources(for a
> > > > single
> > > > machine):
> > > > CPU: 700% / 1600% (16
> cores)
> > > > MEM: almost 100% (16GB)
> > > > Swap: almost 0%
> > > > Disk IO(write):
> 20~30MB / 200MB
> > (7.2k raid5,
> > > benchmarked
> > > > previously)
> > > > NET: up to 100Mbps /
> 950Mbps
> > (1Gbps, tuned and
> > > benchmarked
> > > > previously)
> > > >
> > > > So the speed of
> generating load,
> > about 15M/s as
> > > reported
> > > > before seems
> > > > quite slow to me. I
> assume the
> > system should get at
> > > least
> > > > about 50MB/s
> > > > of Disk IO speed.
> > > >
> > > > MEM? I don't think it
> plays a
> > major role in this
> > > writing game.
> > > > What's
> > > > the bottleneck of the
> system?
> > > >
> > > > P.S
> > > > about Consistency Level,
> I've
> > tried ONE/DCQUORUM and
> > > found ONE
> > > > is about
> > > > 10-15% faster. However
> that's
> > neither a promising
> > > result.
> > > >
> > > > Thanks!
> > > >
> > > > Kevin
> > > >
> > > > >
> > > > > 2010/6/1 史英杰
> > <sh...@gmail.com>:
> > > > > > Hi, It would be
> better if we
> > know which
> > > Consistency Level
> > > > did you choose,
> > > > > > and what is the
> schema of test
> > data?
> > > > > >
> > > > > > 在 2010年6月1日 下午
> 4:48，
> > Shuai Yuan
> > > >
> <yu...@supertool.net.cn>写道：
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I'm testing writing
> speed of
> > cassandra with 4
> > > servers.
> > > > I'm confused by
> > > > > >> the behavior of
> cassandra.
> > > > > >>
> > > > > >> ---env---
> > > > > >> load-data app
> written in c++,
> > using
> > > libcassandra (w/
> > > > modified batch
> > > > > >> insert)
> > > > > >> 20 writing threads
> in 2
> > processes running on 2
> > > servers
> > > > > >>
> > > > > >> ---optimization---
> > > > > >> 1.turn log level to
> INFO
> > > > > >> 2.JVM has 8G heap
> > > > > >> 3.32 concurrent
> read & 128
> > write in
> > > storage-conf.xml,
> > > > other cache
> > > > > >> enlarged as well.
> > > > > >>
> > > > > >> ---result---
> > > > > >> 1-monitoring by
> > `date;nodetool -h host ring`
> > > > > >> I add all load
> together and
> > measure the writing
> > > speed by
> > > > > >> (load_difference /
> > time_difference), and I get
> > > about
> > > > 15MB/s for the
> > > > > >> whole cluster.
> > > > > >>
> > > > > >> 2-monitoring by
> `iostat -m
> > 10`
> > > > > >> I can watch the
> disk_io from
> > the system level
> > > and have
> > > > about 10MB/s -
> > > > > >> 65MB/s for a single
> machine.
> > Very big variance
> > > over time.
> > > > > >>
> > > > > >> 3-monitoring by
> `iptraf -g`
> > > > > >> In this way I watch
> the
> > communication between
> > > servers and
> > > > get about
> > > > > >> 10MB/s for a single
> machine.
> > > > > >>
> > > > > >> ---opinion---
> > > > > >> So, have you
> checked the
> > writing speed of
> > > cassandra? I
> > > > feel it's quite
> > > > > >> slow currently.
> > > > > >>
> > > > > >> Could anyone
> confirm this is
> > the normal writing
> > > speed of
> > > > cassandra, or
> > > > > >> please provide
> someway of
> > improving it?
> > > > > >> --
> > > > > >> Kevin Yuan
> > > > > >> www.yuan-shuai.info
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Kevin Yuan
> > > > www.yuan-shuai.info
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shuai Yuan 袁帅
> > > Supertool Corp. 北京学之途网络科技
> 有限公司
> > >
> > > www.yuan-shuai.info
> > >
> > >
> > >
> >
> >
> > --
> >
> > Shuai Yuan 袁帅
> > Supertool Corp. 北京学之途网络科技有限公司
> > www.yuan-shuai.info
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
> www.yuan-shuai.info
>
>
>
>
>
--
Kevin Yuan
www.yuan-shuai.info
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing
speed test
Posted by lwl <lw...@gmail.com>.
No.
But I did some capacity tests about another distributed system.
Your former test cost too much MEM, it was the bottleneck.
caches and JVM cost MEM, so I suggested to decrease them.
What is the bottleneck of your current test now?
在 2010年6月2日 下午4:13，Shuai Yuan <yu...@supertool.net.cn>写道：
> Hi,
>
> I tried,
>
> 1-consistency level ZERO
>
> 2-JVM heap 4GB
>
> 3-normal Memtable cache
>
> and now I have about 30% improvment.
>
> However I want to know if you have also done w/r benchmark and what's
> the result?
>
> 在 2010-06-02三的 11:35 +0800，lwl写道：
> > and, why did you set "JVM has 8G heap"?
> > 8g, seems too big.
> >
> > 在 2010年6月2日 上午11:20，lwl <lw...@gmail.com>写道：
> > 3.32 concurrent read & 128 write in storage-conf.xml, other
> > cache
> > enlarged as well.
> > --------------------
> >
> >
> > maybe you can try to decrease the size of caches.
> >
> > 在 2010年6月2日 上午11:14，Shuai Yuan
> > <yu...@supertool.net.cn>写道：
> >
> >
> > 在 2010-06-02三的 10:37 +0800，lwl写道：
> > > is all the 4 servers' MEM almost 100%?
> >
> >
> > Yes
> >
> >
> > > 在 2010年6月2日 上午10:12，Shuai Yuan
> > <yu...@supertool.net.cn>写
> > > 道：
> > > Thanks lwl.
> > >
> > > Then is there anyway of tuning this, faster
> > flush to disk or
> > > else?
> > >
> > > Cheers,
> > >
> > > Kevin
> > >
> > > 在 2010-06-02三的 09:57 +0800，lwl写道：
> > >
> > > > MEM: almost 100% (16GB)
> > > > -----------------
> > > > maybe this is the bottleneck.
> > > > writing concerns Memtable and SSTable in
> > memory.
> > > >
> > > > 在 2010年6月2日 上午9:48，Shuai Yuan
> > > <yu...@supertool.net.cn>写
> > > > 道：
> > > > 在 2010-06-01二的 15:00 -0500，
> > Jonathan Shook写道：
> > > > > Also, what are you meaning
> > specifically by 'slow'?
> > > Which
> > > > measurements
> > > > > are you looking at. What are
> > your baseline
> > > constraints for
> > > > your test
> > > > > system?
> > > > >
> > > >
> > > > Actually, the problem is the
> > utilizaton of
> > > resources(for a
> > > > single
> > > > machine):
> > > > CPU: 700% / 1600% (16 cores)
> > > > MEM: almost 100% (16GB)
> > > > Swap: almost 0%
> > > > Disk IO(write): 20~30MB / 200MB
> > (7.2k raid5,
> > > benchmarked
> > > > previously)
> > > > NET: up to 100Mbps / 950Mbps
> > (1Gbps, tuned and
> > > benchmarked
> > > > previously)
> > > >
> > > > So the speed of generating load,
> > about 15M/s as
> > > reported
> > > > before seems
> > > > quite slow to me. I assume the
> > system should get at
> > > least
> > > > about 50MB/s
> > > > of Disk IO speed.
> > > >
> > > > MEM? I don't think it plays a
> > major role in this
> > > writing game.
> > > > What's
> > > > the bottleneck of the system?
> > > >
> > > > P.S
> > > > about Consistency Level, I've
> > tried ONE/DCQUORUM and
> > > found ONE
> > > > is about
> > > > 10-15% faster. However that's
> > neither a promising
> > > result.
> > > >
> > > > Thanks!
> > > >
> > > > Kevin
> > > >
> > > > >
> > > > > 2010/6/1 史英杰
> > <sh...@gmail.com>:
> > > > > > Hi, It would be better if we
> > know which
> > > Consistency Level
> > > > did you choose,
> > > > > > and what is the schema of test
> > data?
> > > > > >
> > > > > > 在 2010年6月1日 下午4:48，
> > Shuai Yuan
> > > > <yu...@supertool.net.cn>写道：
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >> I'm testing writing speed of
> > cassandra with 4
> > > servers.
> > > > I'm confused by
> > > > > >> the behavior of cassandra.
> > > > > >>
> > > > > >> ---env---
> > > > > >> load-data app written in c++,
> > using
> > > libcassandra (w/
> > > > modified batch
> > > > > >> insert)
> > > > > >> 20 writing threads in 2
> > processes running on 2
> > > servers
> > > > > >>
> > > > > >> ---optimization---
> > > > > >> 1.turn log level to INFO
> > > > > >> 2.JVM has 8G heap
> > > > > >> 3.32 concurrent read & 128
> > write in
> > > storage-conf.xml,
> > > > other cache
> > > > > >> enlarged as well.
> > > > > >>
> > > > > >> ---result---
> > > > > >> 1-monitoring by
> > `date;nodetool -h host ring`
> > > > > >> I add all load together and
> > measure the writing
> > > speed by
> > > > > >> (load_difference /
> > time_difference), and I get
> > > about
> > > > 15MB/s for the
> > > > > >> whole cluster.
> > > > > >>
> > > > > >> 2-monitoring by `iostat -m
> > 10`
> > > > > >> I can watch the disk_io from
> > the system level
> > > and have
> > > > about 10MB/s -
> > > > > >> 65MB/s for a single machine.
> > Very big variance
> > > over time.
> > > > > >>
> > > > > >> 3-monitoring by `iptraf -g`
> > > > > >> In this way I watch the
> > communication between
> > > servers and
> > > > get about
> > > > > >> 10MB/s for a single machine.
> > > > > >>
> > > > > >> ---opinion---
> > > > > >> So, have you checked the
> > writing speed of
> > > cassandra? I
> > > > feel it's quite
> > > > > >> slow currently.
> > > > > >>
> > > > > >> Could anyone confirm this is
> > the normal writing
> > > speed of
> > > > cassandra, or
> > > > > >> please provide someway of
> > improving it?
> > > > > >> --
> > > > > >> Kevin Yuan
> > > > > >> www.yuan-shuai.info
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Kevin Yuan
> > > > www.yuan-shuai.info
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shuai Yuan 袁帅
> > > Supertool Corp. 北京学之途网络科技有限公司
> > >
> > > www.yuan-shuai.info
> > >
> > >
> > >
> >
> >
> > --
> >
> > Shuai Yuan 袁帅
> > Supertool Corp. 北京学之途网络科技有限公司
> > www.yuan-shuai.info
> >
> >
> >
> >
> >
> >
> >
>
> --
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
> www.yuan-shuai.info
>
>
>
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing
speed test
Posted by Shuai Yuan <yu...@supertool.net.cn>.
Hi,
I tried,
1-consistency level ZERO
2-JVM heap 4GB
3-normal Memtable cache
and now I have about 30% improvment.
However I want to know if you have also done w/r benchmark and what's
the result?
在 2010-06-02三的 11:35 +0800，lwl写道：
> and, why did you set "JVM has 8G heap"?
> 8g, seems too big.
>
> 在 2010年6月2日 上午11:20，lwl <lw...@gmail.com>写道：
> 3.32 concurrent read & 128 write in storage-conf.xml, other
> cache
> enlarged as well.
> --------------------
>
>
> maybe you can try to decrease the size of caches.
>
> 在 2010年6月2日 上午11:14，Shuai Yuan
> <yu...@supertool.net.cn>写道：
>
>
> 在 2010-06-02三的 10:37 +0800，lwl写道：
> > is all the 4 servers' MEM almost 100%?
>
>
> Yes
>
>
> > 在 2010年6月2日 上午10:12，Shuai Yuan
> <yu...@supertool.net.cn>写
> > 道：
> > Thanks lwl.
> >
> > Then is there anyway of tuning this, faster
> flush to disk or
> > else?
> >
> > Cheers,
> >
> > Kevin
> >
> > 在 2010-06-02三的 09:57 +0800，lwl写道：
> >
> > > MEM: almost 100% (16GB)
> > > -----------------
> > > maybe this is the bottleneck.
> > > writing concerns Memtable and SSTable in
> memory.
> > >
> > > 在 2010年6月2日 上午9:48，Shuai Yuan
> > <yu...@supertool.net.cn>写
> > > 道：
> > > 在 2010-06-01二的 15:00 -0500，
> Jonathan Shook写道：
> > > > Also, what are you meaning
> specifically by 'slow'?
> > Which
> > > measurements
> > > > are you looking at. What are
> your baseline
> > constraints for
> > > your test
> > > > system?
> > > >
> > >
> > > Actually, the problem is the
> utilizaton of
> > resources(for a
> > > single
> > > machine):
> > > CPU: 700% / 1600% (16 cores)
> > > MEM: almost 100% (16GB)
> > > Swap: almost 0%
> > > Disk IO(write): 20~30MB / 200MB
> (7.2k raid5,
> > benchmarked
> > > previously)
> > > NET: up to 100Mbps / 950Mbps
> (1Gbps, tuned and
> > benchmarked
> > > previously)
> > >
> > > So the speed of generating load,
> about 15M/s as
> > reported
> > > before seems
> > > quite slow to me. I assume the
> system should get at
> > least
> > > about 50MB/s
> > > of Disk IO speed.
> > >
> > > MEM? I don't think it plays a
> major role in this
> > writing game.
> > > What's
> > > the bottleneck of the system?
> > >
> > > P.S
> > > about Consistency Level, I've
> tried ONE/DCQUORUM and
> > found ONE
> > > is about
> > > 10-15% faster. However that's
> neither a promising
> > result.
> > >
> > > Thanks!
> > >
> > > Kevin
> > >
> > > >
> > > > 2010/6/1 史英杰
> <sh...@gmail.com>:
> > > > > Hi, It would be better if we
> know which
> > Consistency Level
> > > did you choose,
> > > > > and what is the schema of test
> data?
> > > > >
> > > > > 在 2010年6月1日 下午4:48，
> Shuai Yuan
> > > <yu...@supertool.net.cn>写道：
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> I'm testing writing speed of
> cassandra with 4
> > servers.
> > > I'm confused by
> > > > >> the behavior of cassandra.
> > > > >>
> > > > >> ---env---
> > > > >> load-data app written in c++,
> using
> > libcassandra (w/
> > > modified batch
> > > > >> insert)
> > > > >> 20 writing threads in 2
> processes running on 2
> > servers
> > > > >>
> > > > >> ---optimization---
> > > > >> 1.turn log level to INFO
> > > > >> 2.JVM has 8G heap
> > > > >> 3.32 concurrent read & 128
> write in
> > storage-conf.xml,
> > > other cache
> > > > >> enlarged as well.
> > > > >>
> > > > >> ---result---
> > > > >> 1-monitoring by
> `date;nodetool -h host ring`
> > > > >> I add all load together and
> measure the writing
> > speed by
> > > > >> (load_difference /
> time_difference), and I get
> > about
> > > 15MB/s for the
> > > > >> whole cluster.
> > > > >>
> > > > >> 2-monitoring by `iostat -m
> 10`
> > > > >> I can watch the disk_io from
> the system level
> > and have
> > > about 10MB/s -
> > > > >> 65MB/s for a single machine.
> Very big variance
> > over time.
> > > > >>
> > > > >> 3-monitoring by `iptraf -g`
> > > > >> In this way I watch the
> communication between
> > servers and
> > > get about
> > > > >> 10MB/s for a single machine.
> > > > >>
> > > > >> ---opinion---
> > > > >> So, have you checked the
> writing speed of
> > cassandra? I
> > > feel it's quite
> > > > >> slow currently.
> > > > >>
> > > > >> Could anyone confirm this is
> the normal writing
> > speed of
> > > cassandra, or
> > > > >> please provide someway of
> improving it?
> > > > >> --
> > > > >> Kevin Yuan
> > > > >> www.yuan-shuai.info
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Kevin Yuan
> > > www.yuan-shuai.info
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Shuai Yuan 袁帅
> > Supertool Corp. 北京学之途网络科技有限公司
> >
> > www.yuan-shuai.info
> >
> >
> >
>
>
> --
>
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
> www.yuan-shuai.info
>
>
>
>
>
>
>
--
Shuai Yuan 袁帅
Supertool Corp. 北京学之途网络科技有限公司
www.yuan-shuai.info
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test
Posted by lwl <lw...@gmail.com>.
and, why did you set "JVM has 8G heap"?
8g, seems too big.
在 2010年6月2日 上午11:20，lwl <lw...@gmail.com>写道：
> 3.32 concurrent read & 128 write in storage-conf.xml, other cache
> enlarged as well.
> --------------------
>
> maybe you can try to decrease the size of caches.
>
> 在 2010年6月2日 上午11:14，Shuai Yuan <yu...@supertool.net.cn>写道：
>
> 在 2010-06-02三的 10:37 +0800，lwl写道：
>> > is all the 4 servers' MEM almost 100%?
>>
>> Yes
>>
>> > 在 2010年6月2日 上午10:12，Shuai Yuan <yu...@supertool.net.cn>写
>> > 道：
>> > Thanks lwl.
>> >
>> > Then is there anyway of tuning this, faster flush to disk or
>> > else?
>> >
>> > Cheers,
>> >
>> > Kevin
>> >
>> > 在 2010-06-02三的 09:57 +0800，lwl写道：
>> >
>> > > MEM: almost 100% (16GB)
>> > > -----------------
>> > > maybe this is the bottleneck.
>> > > writing concerns Memtable and SSTable in memory.
>> > >
>> > > 在 2010年6月2日 上午9:48，Shuai Yuan
>> > <yu...@supertool.net.cn>写
>> > > 道：
>> > > 在 2010-06-01二的 15:00 -0500，Jonathan Shook写道：
>> > > > Also, what are you meaning specifically by 'slow'?
>> > Which
>> > > measurements
>> > > > are you looking at. What are your baseline
>> > constraints for
>> > > your test
>> > > > system?
>> > > >
>> > >
>> > > Actually, the problem is the utilizaton of
>> > resources(for a
>> > > single
>> > > machine):
>> > > CPU: 700% / 1600% (16 cores)
>> > > MEM: almost 100% (16GB)
>> > > Swap: almost 0%
>> > > Disk IO(write): 20~30MB / 200MB (7.2k raid5,
>> > benchmarked
>> > > previously)
>> > > NET: up to 100Mbps / 950Mbps (1Gbps, tuned and
>> > benchmarked
>> > > previously)
>> > >
>> > > So the speed of generating load, about 15M/s as
>> > reported
>> > > before seems
>> > > quite slow to me. I assume the system should get at
>> > least
>> > > about 50MB/s
>> > > of Disk IO speed.
>> > >
>> > > MEM? I don't think it plays a major role in this
>> > writing game.
>> > > What's
>> > > the bottleneck of the system?
>> > >
>> > > P.S
>> > > about Consistency Level, I've tried ONE/DCQUORUM and
>> > found ONE
>> > > is about
>> > > 10-15% faster. However that's neither a promising
>> > result.
>> > >
>> > > Thanks!
>> > >
>> > > Kevin
>> > >
>> > > >
>> > > > 2010/6/1 史英杰 <sh...@gmail.com>:
>> > > > > Hi, It would be better if we know which
>> > Consistency Level
>> > > did you choose,
>> > > > > and what is the schema of test data?
>> > > > >
>> > > > > 在 2010年6月1日 下午4:48，Shuai Yuan
>> > > <yu...@supertool.net.cn>写道：
>> > > > >>
>> > > > >> Hi all,
>> > > > >>
>> > > > >> I'm testing writing speed of cassandra with 4
>> > servers.
>> > > I'm confused by
>> > > > >> the behavior of cassandra.
>> > > > >>
>> > > > >> ---env---
>> > > > >> load-data app written in c++, using
>> > libcassandra (w/
>> > > modified batch
>> > > > >> insert)
>> > > > >> 20 writing threads in 2 processes running on 2
>> > servers
>> > > > >>
>> > > > >> ---optimization---
>> > > > >> 1.turn log level to INFO
>> > > > >> 2.JVM has 8G heap
>> > > > >> 3.32 concurrent read & 128 write in
>> > storage-conf.xml,
>> > > other cache
>> > > > >> enlarged as well.
>> > > > >>
>> > > > >> ---result---
>> > > > >> 1-monitoring by `date;nodetool -h host ring`
>> > > > >> I add all load together and measure the writing
>> > speed by
>> > > > >> (load_difference / time_difference), and I get
>> > about
>> > > 15MB/s for the
>> > > > >> whole cluster.
>> > > > >>
>> > > > >> 2-monitoring by `iostat -m 10`
>> > > > >> I can watch the disk_io from the system level
>> > and have
>> > > about 10MB/s -
>> > > > >> 65MB/s for a single machine. Very big variance
>> > over time.
>> > > > >>
>> > > > >> 3-monitoring by `iptraf -g`
>> > > > >> In this way I watch the communication between
>> > servers and
>> > > get about
>> > > > >> 10MB/s for a single machine.
>> > > > >>
>> > > > >> ---opinion---
>> > > > >> So, have you checked the writing speed of
>> > cassandra? I
>> > > feel it's quite
>> > > > >> slow currently.
>> > > > >>
>> > > > >> Could anyone confirm this is the normal writing
>> > speed of
>> > > cassandra, or
>> > > > >> please provide someway of improving it?
>> > > > >> --
>> > > > >> Kevin Yuan
>> > > > >> www.yuan-shuai.info
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Kevin Yuan
>> > > www.yuan-shuai.info
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Shuai Yuan 袁帅
>> > Supertool Corp. 北京学之途网络科技有限公司
>> >
>> > www.yuan-shuai.info
>> >
>> >
>> >
>>
>> --
>> Shuai Yuan 袁帅
>> Supertool Corp. 北京学之途网络科技有限公司
>> www.yuan-shuai.info
>>
>>
>>
>
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test
Posted by lwl <lw...@gmail.com>.
3.32 concurrent read & 128 write in storage-conf.xml, other cache
enlarged as well.
--------------------
maybe you can try to decrease the size of caches.
在 2010年6月2日 上午11:14，Shuai Yuan <yu...@supertool.net.cn>写道：
> 在 2010-06-02三的 10:37 +0800，lwl写道：
> > is all the 4 servers' MEM almost 100%?
>
> Yes
>
> > 在 2010年6月2日 上午10:12，Shuai Yuan <yu...@supertool.net.cn>写
> > 道：
> > Thanks lwl.
> >
> > Then is there anyway of tuning this, faster flush to disk or
> > else?
> >
> > Cheers,
> >
> > Kevin
> >
> > 在 2010-06-02三的 09:57 +0800，lwl写道：
> >
> > > MEM: almost 100% (16GB)
> > > -----------------
> > > maybe this is the bottleneck.
> > > writing concerns Memtable and SSTable in memory.
> > >
> > > 在 2010年6月2日 上午9:48，Shuai Yuan
> > <yu...@supertool.net.cn>写
> > > 道：
> > > 在 2010-06-01二的 15:00 -0500，Jonathan Shook写道：
> > > > Also, what are you meaning specifically by 'slow'?
> > Which
> > > measurements
> > > > are you looking at. What are your baseline
> > constraints for
> > > your test
> > > > system?
> > > >
> > >
> > > Actually, the problem is the utilizaton of
> > resources(for a
> > > single
> > > machine):
> > > CPU: 700% / 1600% (16 cores)
> > > MEM: almost 100% (16GB)
> > > Swap: almost 0%
> > > Disk IO(write): 20~30MB / 200MB (7.2k raid5,
> > benchmarked
> > > previously)
> > > NET: up to 100Mbps / 950Mbps (1Gbps, tuned and
> > benchmarked
> > > previously)
> > >
> > > So the speed of generating load, about 15M/s as
> > reported
> > > before seems
> > > quite slow to me. I assume the system should get at
> > least
> > > about 50MB/s
> > > of Disk IO speed.
> > >
> > > MEM? I don't think it plays a major role in this
> > writing game.
> > > What's
> > > the bottleneck of the system?
> > >
> > > P.S
> > > about Consistency Level, I've tried ONE/DCQUORUM and
> > found ONE
> > > is about
> > > 10-15% faster. However that's neither a promising
> > result.
> > >
> > > Thanks!
> > >
> > > Kevin
> > >
> > > >
> > > > 2010/6/1 史英杰 <sh...@gmail.com>:
> > > > > Hi, It would be better if we know which
> > Consistency Level
> > > did you choose,
> > > > > and what is the schema of test data?
> > > > >
> > > > > 在 2010年6月1日 下午4:48，Shuai Yuan
> > > <yu...@supertool.net.cn>写道：
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> I'm testing writing speed of cassandra with 4
> > servers.
> > > I'm confused by
> > > > >> the behavior of cassandra.
> > > > >>
> > > > >> ---env---
> > > > >> load-data app written in c++, using
> > libcassandra (w/
> > > modified batch
> > > > >> insert)
> > > > >> 20 writing threads in 2 processes running on 2
> > servers
> > > > >>
> > > > >> ---optimization---
> > > > >> 1.turn log level to INFO
> > > > >> 2.JVM has 8G heap
> > > > >> 3.32 concurrent read & 128 write in
> > storage-conf.xml,
> > > other cache
> > > > >> enlarged as well.
> > > > >>
> > > > >> ---result---
> > > > >> 1-monitoring by `date;nodetool -h host ring`
> > > > >> I add all load together and measure the writing
> > speed by
> > > > >> (load_difference / time_difference), and I get
> > about
> > > 15MB/s for the
> > > > >> whole cluster.
> > > > >>
> > > > >> 2-monitoring by `iostat -m 10`
> > > > >> I can watch the disk_io from the system level
> > and have
> > > about 10MB/s -
> > > > >> 65MB/s for a single machine. Very big variance
> > over time.
> > > > >>
> > > > >> 3-monitoring by `iptraf -g`
> > > > >> In this way I watch the communication between
> > servers and
> > > get about
> > > > >> 10MB/s for a single machine.
> > > > >>
> > > > >> ---opinion---
> > > > >> So, have you checked the writing speed of
> > cassandra? I
> > > feel it's quite
> > > > >> slow currently.
> > > > >>
> > > > >> Could anyone confirm this is the normal writing
> > speed of
> > > cassandra, or
> > > > >> please provide someway of improving it?
> > > > >> --
> > > > >> Kevin Yuan
> > > > >> www.yuan-shuai.info
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Kevin Yuan
> > > www.yuan-shuai.info
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Shuai Yuan 袁帅
> > Supertool Corp. 北京学之途网络科技有限公司
> >
> > www.yuan-shuai.info
> >
> >
> >
>
> --
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
> www.yuan-shuai.info
>
>
>
Re: [***SPAM*** ] Re: [***SPAM*** ] Re: writing speed test
Posted by Shuai Yuan <yu...@supertool.net.cn>.
在 2010-06-02三的 10:37 +0800，lwl写道：
> is all the 4 servers' MEM almost 100%?
Yes
> 在 2010年6月2日 上午10:12，Shuai Yuan <yu...@supertool.net.cn>写
> 道：
> Thanks lwl.
>
> Then is there anyway of tuning this, faster flush to disk or
> else?
>
> Cheers,
>
> Kevin
>
> 在 2010-06-02三的 09:57 +0800，lwl写道：
>
> > MEM: almost 100% (16GB)
> > -----------------
> > maybe this is the bottleneck.
> > writing concerns Memtable and SSTable in memory.
> >
> > 在 2010年6月2日 上午9:48，Shuai Yuan
> <yu...@supertool.net.cn>写
> > 道：
> > 在 2010-06-01二的 15:00 -0500，Jonathan Shook写道：
> > > Also, what are you meaning specifically by 'slow'?
> Which
> > measurements
> > > are you looking at. What are your baseline
> constraints for
> > your test
> > > system?
> > >
> >
> > Actually, the problem is the utilizaton of
> resources(for a
> > single
> > machine):
> > CPU: 700% / 1600% (16 cores)
> > MEM: almost 100% (16GB)
> > Swap: almost 0%
> > Disk IO(write): 20~30MB / 200MB (7.2k raid5,
> benchmarked
> > previously)
> > NET: up to 100Mbps / 950Mbps (1Gbps, tuned and
> benchmarked
> > previously)
> >
> > So the speed of generating load, about 15M/s as
> reported
> > before seems
> > quite slow to me. I assume the system should get at
> least
> > about 50MB/s
> > of Disk IO speed.
> >
> > MEM? I don't think it plays a major role in this
> writing game.
> > What's
> > the bottleneck of the system?
> >
> > P.S
> > about Consistency Level, I've tried ONE/DCQUORUM and
> found ONE
> > is about
> > 10-15% faster. However that's neither a promising
> result.
> >
> > Thanks!
> >
> > Kevin
> >
> > >
> > > 2010/6/1 史英杰 <sh...@gmail.com>:
> > > > Hi, It would be better if we know which
> Consistency Level
> > did you choose,
> > > > and what is the schema of test data?
> > > >
> > > > 在 2010年6月1日 下午4:48，Shuai Yuan
> > <yu...@supertool.net.cn>写道：
> > > >>
> > > >> Hi all,
> > > >>
> > > >> I'm testing writing speed of cassandra with 4
> servers.
> > I'm confused by
> > > >> the behavior of cassandra.
> > > >>
> > > >> ---env---
> > > >> load-data app written in c++, using
> libcassandra (w/
> > modified batch
> > > >> insert)
> > > >> 20 writing threads in 2 processes running on 2
> servers
> > > >>
> > > >> ---optimization---
> > > >> 1.turn log level to INFO
> > > >> 2.JVM has 8G heap
> > > >> 3.32 concurrent read & 128 write in
> storage-conf.xml,
> > other cache
> > > >> enlarged as well.
> > > >>
> > > >> ---result---
> > > >> 1-monitoring by `date;nodetool -h host ring`
> > > >> I add all load together and measure the writing
> speed by
> > > >> (load_difference / time_difference), and I get
> about
> > 15MB/s for the
> > > >> whole cluster.
> > > >>
> > > >> 2-monitoring by `iostat -m 10`
> > > >> I can watch the disk_io from the system level
> and have
> > about 10MB/s -
> > > >> 65MB/s for a single machine. Very big variance
> over time.
> > > >>
> > > >> 3-monitoring by `iptraf -g`
> > > >> In this way I watch the communication between
> servers and
> > get about
> > > >> 10MB/s for a single machine.
> > > >>
> > > >> ---opinion---
> > > >> So, have you checked the writing speed of
> cassandra? I
> > feel it's quite
> > > >> slow currently.
> > > >>
> > > >> Could anyone confirm this is the normal writing
> speed of
> > cassandra, or
> > > >> please provide someway of improving it?
> > > >> --
> > > >> Kevin Yuan
> > > >> www.yuan-shuai.info
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > Kevin Yuan
> > www.yuan-shuai.info
> >
> >
> >
> >
> >
>
>
> --
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
>
> www.yuan-shuai.info
>
>
>
--
Shuai Yuan 袁帅
Supertool Corp. 北京学之途网络科技有限公司
www.yuan-shuai.info
Re: [***SPAM*** ] Re: writing speed test
Posted by lwl <lw...@gmail.com>.
is all the 4 servers' MEM almost 100%?
在 2010年6月2日 上午10:12，Shuai Yuan <yu...@supertool.net.cn>写道：
> Thanks lwl.
>
> Then is there anyway of tuning this, faster flush to disk or else?
>
> Cheers,
>
> Kevin
>
> 在 2010-06-02三的 09:57 +0800，lwl写道：
> > MEM: almost 100% (16GB)
> > -----------------
> > maybe this is the bottleneck.
> > writing concerns Memtable and SSTable in memory.
> >
> > 在 2010年6月2日 上午9:48，Shuai Yuan <yu...@supertool.net.cn>写
> > 道：
> > 在 2010-06-01二的 15:00 -0500，Jonathan Shook写道：
> > > Also, what are you meaning specifically by 'slow'? Which
> > measurements
> > > are you looking at. What are your baseline constraints for
> > your test
> > > system?
> > >
> >
> > Actually, the problem is the utilizaton of resources(for a
> > single
> > machine):
> > CPU: 700% / 1600% (16 cores)
> > MEM: almost 100% (16GB)
> > Swap: almost 0%
> > Disk IO(write): 20~30MB / 200MB (7.2k raid5, benchmarked
> > previously)
> > NET: up to 100Mbps / 950Mbps (1Gbps, tuned and benchmarked
> > previously)
> >
> > So the speed of generating load, about 15M/s as reported
> > before seems
> > quite slow to me. I assume the system should get at least
> > about 50MB/s
> > of Disk IO speed.
> >
> > MEM? I don't think it plays a major role in this writing game.
> > What's
> > the bottleneck of the system?
> >
> > P.S
> > about Consistency Level, I've tried ONE/DCQUORUM and found ONE
> > is about
> > 10-15% faster. However that's neither a promising result.
> >
> > Thanks!
> >
> > Kevin
> >
> > >
> > > 2010/6/1 史英杰 <sh...@gmail.com>:
> > > > Hi, It would be better if we know which Consistency Level
> > did you choose,
> > > > and what is the schema of test data?
> > > >
> > > > 在 2010年6月1日 下午4:48，Shuai Yuan
> > <yu...@supertool.net.cn>写道：
> > > >>
> > > >> Hi all,
> > > >>
> > > >> I'm testing writing speed of cassandra with 4 servers.
> > I'm confused by
> > > >> the behavior of cassandra.
> > > >>
> > > >> ---env---
> > > >> load-data app written in c++, using libcassandra (w/
> > modified batch
> > > >> insert)
> > > >> 20 writing threads in 2 processes running on 2 servers
> > > >>
> > > >> ---optimization---
> > > >> 1.turn log level to INFO
> > > >> 2.JVM has 8G heap
> > > >> 3.32 concurrent read & 128 write in storage-conf.xml,
> > other cache
> > > >> enlarged as well.
> > > >>
> > > >> ---result---
> > > >> 1-monitoring by `date;nodetool -h host ring`
> > > >> I add all load together and measure the writing speed by
> > > >> (load_difference / time_difference), and I get about
> > 15MB/s for the
> > > >> whole cluster.
> > > >>
> > > >> 2-monitoring by `iostat -m 10`
> > > >> I can watch the disk_io from the system level and have
> > about 10MB/s -
> > > >> 65MB/s for a single machine. Very big variance over time.
> > > >>
> > > >> 3-monitoring by `iptraf -g`
> > > >> In this way I watch the communication between servers and
> > get about
> > > >> 10MB/s for a single machine.
> > > >>
> > > >> ---opinion---
> > > >> So, have you checked the writing speed of cassandra? I
> > feel it's quite
> > > >> slow currently.
> > > >>
> > > >> Could anyone confirm this is the normal writing speed of
> > cassandra, or
> > > >> please provide someway of improving it?
> > > >> --
> > > >> Kevin Yuan
> > > >> www.yuan-shuai.info
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > Kevin Yuan
> > www.yuan-shuai.info
> >
> >
> >
> >
> >
>
> --
> Shuai Yuan 袁帅
> Supertool Corp. 北京学之途网络科技有限公司
> www.yuan-shuai.info
>
>
>