You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by pob <pe...@gmail.com> on 2011/03/21 12:02:17 UTC

stress.py bug?

Hi,

I'm inserting data from client node with stress.py to cluster of 6 nodes.
They are all on 1Gbps network, max real throughput of network is 930Mbps
(after measurement).

python stress.py -c 1 -S 170000  -d{6nodes}  -l3 -e QUORUM
 --operation=insert -i 1 -n 500000 -t100

The problem is stress.py show up it does avg ~750ops/sec what is 127MB/s,
but the real throughput of network is ~116MB/s.


Any idea?



Thanks


Best,
Peter

Re: stress.py bug?

Posted by A J <s5...@gmail.com>.
Not completely related. just fyi.

I like it better to see the start time, end time, duration of each
execution in each thread. And then do the aggregation (avg,max,min)
myself.

I modified last few lines of the Inserter function as follows:
endtime = time.time()
            self.latencies[self.idx] += endtime - start
            self.opcounts[self.idx] += 1
            self.keycounts[self.idx] += 1
            open('log'+str(self.idx)+'.txt','a').write(str(endtime-start)
+ ' ' + str(self.idx) + ' ' + str(i) + ' ' + str(time.asctime())+ ' '
+ str(start) + ' ' + str(endtime) + '\n')

You need to understand little bit of python to plug this properly in stress.py.

Above creates lot of log*.txt files. One for each thread.
Each line in these log files have the duration,
thread#,key,timestamp,starttime,endtime separated by space.

i then load these log files to a database and do aggregations as I need.

Remember to remove the old log files on rerun. The above will append
to existing log files.

Just a fyi. Most will not need this.


On Mon, Mar 21, 2011 at 12:40 PM, Ryan King <ry...@twitter.com> wrote:
> On Mon, Mar 21, 2011 at 9:34 AM, pob <pe...@gmail.com> wrote:
>> You mean,
>> more threads in stress.py? The purpose was figure out whats the
>> biggest bandwidth that C* can use.
>
> You should try more threads, but at some point you'll hit diminishing
> returns there. You many need to drive load from more than one host.
> Either way, you need to find out what the bottleneck is.
>
> -ryan
>

Re: stress.py bug?

Posted by Ryan King <ry...@twitter.com>.
On Mon, Mar 21, 2011 at 9:34 AM, pob <pe...@gmail.com> wrote:
> You mean,
> more threads in stress.py? The purpose was figure out whats the
> biggest bandwidth that C* can use.

You should try more threads, but at some point you'll hit diminishing
returns there. You many need to drive load from more than one host.
Either way, you need to find out what the bottleneck is.

-ryan

Re: stress.py bug?

Posted by pob <pe...@gmail.com>.
You mean,

more threads in stress.py? The purpose was figure out whats the
biggest bandwidth that C* can use.


Peter

2011/3/21 Ryan King <ry...@twitter.com>

> On Mon, Mar 21, 2011 at 4:02 AM, pob <pe...@gmail.com> wrote:
> > Hi,
> > I'm inserting data from client node with stress.py to cluster of 6 nodes.
> > They are all on 1Gbps network, max real throughput of network is 930Mbps
> > (after measurement).
> > python stress.py -c 1 -S 170000  -d{6nodes}  -l3 -e QUORUM
> >  --operation=insert -i 1 -n 500000 -t100
> > The problem is stress.py show up it does avg ~750ops/sec what is 127MB/s,
> > but the real throughput of network is ~116MB/s.
>
> You may need more concurrency in order to saturate your network.
>
> -ryan
>

Re: stress.py bug?

Posted by Maki Watanabe <wa...@gmail.com>.
A client thread need to wait for response, during the server can
handle multiple requests simultaneously.

2011/3/22 Sheng Chen <ch...@gmail.com>:
> I am just wondering, why the stress test tools (python, java) need more
> threads ?
> Is the bottleneck of a single thread in the client, or in the server?
> Thanks.
> Sean
>
> 2011/3/22 Ryan King <ry...@twitter.com>
>>
>> On Mon, Mar 21, 2011 at 4:02 AM, pob <pe...@gmail.com> wrote:
>> > Hi,
>> > I'm inserting data from client node with stress.py to cluster of 6
>> > nodes.
>> > They are all on 1Gbps network, max real throughput of network is 930Mbps
>> > (after measurement).
>> > python stress.py -c 1 -S 170000  -d{6nodes}  -l3 -e QUORUM
>> >  --operation=insert -i 1 -n 500000 -t100
>> > The problem is stress.py show up it does avg ~750ops/sec what is
>> > 127MB/s,
>> > but the real throughput of network is ~116MB/s.
>>
>> You may need more concurrency in order to saturate your network.
>>
>> -ryan
>
>



-- 
w3m

Re: stress.py bug?

Posted by Sheng Chen <ch...@gmail.com>.
I am just wondering, why the stress test tools (python, java) need more
threads ?
Is the bottleneck of a single thread in the client, or in the server?
Thanks.

Sean

2011/3/22 Ryan King <ry...@twitter.com>

> On Mon, Mar 21, 2011 at 4:02 AM, pob <pe...@gmail.com> wrote:
> > Hi,
> > I'm inserting data from client node with stress.py to cluster of 6 nodes.
> > They are all on 1Gbps network, max real throughput of network is 930Mbps
> > (after measurement).
> > python stress.py -c 1 -S 170000  -d{6nodes}  -l3 -e QUORUM
> >  --operation=insert -i 1 -n 500000 -t100
> > The problem is stress.py show up it does avg ~750ops/sec what is 127MB/s,
> > but the real throughput of network is ~116MB/s.
>
> You may need more concurrency in order to saturate your network.
>
> -ryan
>

Re: stress.py bug?

Posted by Ryan King <ry...@twitter.com>.
On Mon, Mar 21, 2011 at 4:02 AM, pob <pe...@gmail.com> wrote:
> Hi,
> I'm inserting data from client node with stress.py to cluster of 6 nodes.
> They are all on 1Gbps network, max real throughput of network is 930Mbps
> (after measurement).
> python stress.py -c 1 -S 170000  -d{6nodes}  -l3 -e QUORUM
>  --operation=insert -i 1 -n 500000 -t100
> The problem is stress.py show up it does avg ~750ops/sec what is 127MB/s,
> but the real throughput of network is ~116MB/s.

You may need more concurrency in order to saturate your network.

-ryan