You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Heath Oderman <he...@526valley.com> on 2010/04/14 06:38:44 UTC
New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Hi,
I wrote a few days ago and got a few good suggestions. I'm still seeing
dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
I've tried on Debian with the Sun JRE and the Open JDK with nearly identical
results. I've tried a mix of hardware.
Attached are some graphs I've produced of my results which show that in OSX,
Cassandra takes longer with a greater load but is wicked fast (expected).
In the SunJDK or Open JDK on Debian I get amazingly consistent time taken to
do the writes, regardless of the load and the times are always ridiculously
high. It's insanely slow.
I genuinely believe that I must be doing something very wrong in my Debian
setups, but they are all vanilla installs, both 64 bit and 32 bit machines,
64bit and 32 bit installs. Cassandra packs taken from
http://www.apache.org/dist/cassandra/debian.
I am using Thrift, and I'm using a c# client because that's how I intend to
actually use Cassandra and it seems pretty sensible.
An example of what I'm seeing is:
5 Threads Each writing 100,000 Simple Entries
OSX: 1 min 16 seconds ~ 6515 Entries / second
Debian: 1 hour 15 seconds ~ 138 Records / second
15 Threads Each writing 100,000 Simple Entries
OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
Debian: 1 hour 1.5 minutes ~406 Entries / second
20 Threads Each Writing 100,000 Simple Entries
OSX: 3min 19 seconds ~ 10,050 Entries / second
Debian: 1 hour 20 seconds ~ 492 Entries / second
If anyone has any suggestions or pointers I'd be glad to hear them.
Thanks,
Stu
Attached:
1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
downloaded from Google Docs)
2. OSX Records per Second - a graph of how many entries get written per
second for 10,000 & 100,000 entries as thread count is increased in OSX.
3. Open JDK Records per Second - the same graph but of Open JDK on Debian
4. Open JDK Total Time By Thread - the total time taken from test start to
finish (all threads completed) to write 10,000 & 100,000 entries as thread
count is increased in Debian with Open JDK
5. OSX Total time by Thread - same as 4, but for OSX.
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Zhiguo Zhang <mi...@gmail.com>.
Hi,
sorry I can't help you, but could you please tell me, how could you get the
charts in the attachment?
Thanks.
Mike
On Wed, Apr 14, 2010 at 6:38 AM, Heath Oderman <he...@526valley.com> wrote:
> Hi,
>
> I wrote a few days ago and got a few good suggestions. I'm still seeing
> dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
>
> I've tried on Debian with the Sun JRE and the Open JDK with nearly
> identical results. I've tried a mix of hardware.
>
> Attached are some graphs I've produced of my results which show that in
> OSX, Cassandra takes longer with a greater load but is wicked fast
> (expected).
>
> In the SunJDK or Open JDK on Debian I get amazingly consistent time taken
> to do the writes, regardless of the load and the times are always
> ridiculously high. It's insanely slow.
>
> I genuinely believe that I must be doing something very wrong in my Debian
> setups, but they are all vanilla installs, both 64 bit and 32 bit machines,
> 64bit and 32 bit installs. Cassandra packs taken from
> http://www.apache.org/dist/cassandra/debian.
>
> I am using Thrift, and I'm using a c# client because that's how I intend to
> actually use Cassandra and it seems pretty sensible.
>
> An example of what I'm seeing is:
>
> 5 Threads Each writing 100,000 Simple Entries
> OSX: 1 min 16 seconds ~ 6515 Entries / second
> Debian: 1 hour 15 seconds ~ 138 Records / second
>
> 15 Threads Each writing 100,000 Simple Entries
> OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
> Debian: 1 hour 1.5 minutes ~406 Entries / second
>
> 20 Threads Each Writing 100,000 Simple Entries
> OSX: 3min 19 seconds ~ 10,050 Entries / second
> Debian: 1 hour 20 seconds ~ 492 Entries / second
>
> If anyone has any suggestions or pointers I'd be glad to hear them.
> Thanks,
> Stu
>
> Attached:
> 1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
> downloaded from Google Docs)
> 2. OSX Records per Second - a graph of how many entries get written per
> second for 10,000 & 100,000 entries as thread count is increased in OSX.
> 3. Open JDK Records per Second - the same graph but of Open JDK on Debian
> 4. Open JDK Total Time By Thread - the total time taken from test start to
> finish (all threads completed) to write 10,000 & 100,000 entries as thread
> count is increased in Debian with Open JDK
> 5. OSX Total time by Thread - same as 4, but for OSX.
>
>
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Brandon Williams <dr...@gmail.com>.
On Fri, Apr 16, 2010 at 10:42 AM, Heath Oderman <he...@526valley.com> wrote:
> Any pointers on why a thrift client would fly against cass on one os vs
> another? The only delta is os.
>
That's probably not the only delta. There's networking involved too, so
perhaps the problem lies there. Let us know how stress.py does remotely.
-Brandon
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Heath Oderman <he...@526valley.com>.
Any pointers on why a thrift client would fly against cass on one os vs
another? The only delta is os.
I'm going to try py_stress remote.
Stu
On Apr 16, 2010 11:39 AM, "Jonathan Ellis" <jb...@gmail.com> wrote:
Sounds like the problem is with the C# client code, then.
On Fri, Apr 16, 2010 at 10:36 AM, Heath Oderman <he...@526valley.com> wrote:
> Ok, it took me a lon...
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like the problem is with the C# client code, then.
On Fri, Apr 16, 2010 at 10:36 AM, Heath Oderman <he...@526valley.com> wrote:
> Ok, it took me a long time to get py_stress working.
> I didn't have thrift / boost / gcc on my debian box :) ....
> I'm using this command line believing it's similar to my c# tests from a
> remote box:
> cnb:~/apache-cassandra-0.6.0-src/contrib/py_stress# python stress.py -o
> insert -n 1000000 -d 10.113.0.195 -y super
> its wicked fast, as you'd expect.
> total,interval_op_rate,avg_latency,elapsed_time
> 61765,6176,0.00804715166591,10
> 126667,6490,0.00763260502319,20
> 184952,5828,0.00798357854098,30
> 249697,6474,0.00813590502387,40
> 310190,6049,0.00820316256729,50
> 370394,6020,0.00823702464729,60
> 431557,6116,0.00811347469409,70
> 492085,6052,0.00818783322119,80
> 558895,6681,0.00740800772112,90
> 620426,6153,0.00805101232533,100
> 685168,6474,0.00766180823714,110
> 748768,6360,0.00779282277485,120
> 811008,6224,0.00797537056523,130
> 867327,5631,0.00881408287019,140
> This leaves me further stumped. I guess i will try running py_stress from a
> remote box, because I've got to believe it has something to do with the
> connection.
> Thanks for nudging me toward py_stress. I'm no closer to understanding, but
> I have more info!
> Stu
> On Thu, Apr 15, 2010 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> What kind of numbers do you get from contrib/py_stress?
>>
>> (that's located somewhere else in 0.5, but you should really be using
>> 0.6 anyway.)
>>
>> On Thu, Apr 15, 2010 at 12:53 PM, Heath Oderman <he...@526valley.com>
>> wrote:
>> > So checking it out quickly:
>> > vmstat -
>> > Never swaps. si and so stay at 0 during the load.
>> > iostat -x
>> > the %util never climbs above 0.00, but the avgrg-sz jumps bewteen
>> > samples
>> > from 0 - 30 - 90 - 0 (5 second intervals)
>> > top shows the cpu barely working and mem utilization is below 20%.
>> > Still slow. :(
>> > Thanks for the suggestions. In your article on your blog it'd be
>> > awesome to
>> > include some implications, like "avgrg-sz over 250 may mean XXX" Even
>> > if
>> > it's utterly hardware and system dependent it'd give a guy like me an
>> > idea
>> > if what I was seeing was bad or good. :D
>> > Thanks again,
>> > Heath
>> >
>> > On Thu, Apr 15, 2010 at 1:34 PM, Heath Oderman <he...@526valley.com>
>> > wrote:
>> >>
>> >> Thanks Jonathan, I'll check this out right away.
>> >>
>> >> On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com>
>> >> wrote:
>> >>>
>> >>> You're right, to get those numbers on debian something is very wrong.
>> >>>
>> >>> Have you looked at
>> >>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
>> >>> What is the bottleneck on the linux machines?
>> >>>
>> >>> With the kind of speed you are seeing I wouldn't be surprised if it is
>> >>> swapping.
>> >>>
>> >>> -Jonathan
>> >>>
>> >>> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
>> >>> wrote:
>> >>> > Hi,
>> >>> > I wrote a few days ago and got a few good suggestions. I'm still
>> >>> > seeing
>> >>> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian
>> >>> > Linux.
>> >>> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
>> >>> > identical
>> >>> > results. I've tried a mix of hardware.
>> >>> > Attached are some graphs I've produced of my results which show that
>> >>> > in
>> >>> > OSX,
>> >>> > Cassandra takes longer with a greater load but is wicked fast
>> >>> > (expected).
>> >>> > In the SunJDK or Open JDK on Debian I get amazingly consistent time
>> >>> > taken to
>> >>> > do the writes, regardless of the load and the times are always
>> >>> > ridiculously
>> >>> > high. It's insanely slow.
>> >>> > I genuinely believe that I must be doing something very wrong in my
>> >>> > Debian
>> >>> > setups, but they are all vanilla installs, both 64 bit and 32 bit
>> >>> > machines,
>> >>> > 64bit and 32 bit installs. Cassandra packs taken from
>> >>> > http://www.apache.org/dist/cassandra/debian.
>> >>> > I am using Thrift, and I'm using a c# client because that's how I
>> >>> > intend to
>> >>> > actually use Cassandra and it seems pretty sensible.
>> >>> > An example of what I'm seeing is:
>> >>> > 5 Threads Each writing 100,000 Simple Entries
>> >>> > OSX: 1 min 16 seconds ~ 6515 Entries / second
>> >>> > Debian: 1 hour 15 seconds ~ 138 Records / second
>> >>> > 15 Threads Each writing 100,000 Simple Entries
>> >>> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
>> >>> > Debian: 1 hour 1.5 minutes ~406 Entries / second
>> >>> > 20 Threads Each Writing 100,000 Simple Entries
>> >>> > OSX: 3min 19 seconds ~ 10,050 Entries / second
>> >>> > Debian: 1 hour 20 seconds ~ 492 Entries / second
>> >>> > If anyone has any suggestions or pointers I'd be glad to hear them.
>> >>> > Thanks,
>> >>> > Stu
>> >>> > Attached:
>> >>> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice
>> >>> > format
>> >>> > downloaded from Google Docs)
>> >>> > 2. OSX Records per Second - a graph of how many entries get written
>> >>> > per
>> >>> > second for 10,000 & 100,000 entries as thread count is increased in
>> >>> > OSX.
>> >>> > 3. Open JDK Records per Second - the same graph but of Open JDK on
>> >>> > Debian
>> >>> > 4. Open JDK Total Time By Thread - the total time taken from test
>> >>> > start
>> >>> > to
>> >>> > finish (all threads completed) to write 10,000 & 100,000 entries as
>> >>> > thread
>> >>> > count is increased in Debian with Open JDK
>> >>> > 5. OSX Total time by Thread - same as 4, but for OSX.
>> >>> >
>> >>> >
>> >>
>> >
>> >
>
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Heath Oderman <he...@526valley.com>.
Ok, it took me a long time to get py_stress working.
I didn't have thrift / boost / gcc on my debian box :) ....
I'm using this command line believing it's similar to my c# tests from a
remote box:
cnb:~/apache-cassandra-0.6.0-src/contrib/py_stress# python stress.py -o
insert -n 1000000 -d 10.113.0.195 -y super
its wicked fast, as you'd expect.
total,interval_op_rate,avg_latency,elapsed_time
61765,6176,0.00804715166591,10
126667,6490,0.00763260502319,20
184952,5828,0.00798357854098,30
249697,6474,0.00813590502387,40
310190,6049,0.00820316256729,50
370394,6020,0.00823702464729,60
431557,6116,0.00811347469409,70
492085,6052,0.00818783322119,80
558895,6681,0.00740800772112,90
620426,6153,0.00805101232533,100
685168,6474,0.00766180823714,110
748768,6360,0.00779282277485,120
811008,6224,0.00797537056523,130
867327,5631,0.00881408287019,140
This leaves me further stumped. I guess i will try running py_stress from a
remote box, because I've got to believe it has something to do with the
connection.
Thanks for nudging me toward py_stress. I'm no closer to understanding, but
I have more info!
Stu
On Thu, Apr 15, 2010 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> What kind of numbers do you get from contrib/py_stress?
>
> (that's located somewhere else in 0.5, but you should really be using
> 0.6 anyway.)
>
> On Thu, Apr 15, 2010 at 12:53 PM, Heath Oderman <he...@526valley.com>
> wrote:
> > So checking it out quickly:
> > vmstat -
> > Never swaps. si and so stay at 0 during the load.
> > iostat -x
> > the %util never climbs above 0.00, but the avgrg-sz jumps bewteen samples
> > from 0 - 30 - 90 - 0 (5 second intervals)
> > top shows the cpu barely working and mem utilization is below 20%.
> > Still slow. :(
> > Thanks for the suggestions. In your article on your blog it'd be awesome
> to
> > include some implications, like "avgrg-sz over 250 may mean XXX" Even if
> > it's utterly hardware and system dependent it'd give a guy like me an
> idea
> > if what I was seeing was bad or good. :D
> > Thanks again,
> > Heath
> >
> > On Thu, Apr 15, 2010 at 1:34 PM, Heath Oderman <he...@526valley.com>
> wrote:
> >>
> >> Thanks Jonathan, I'll check this out right away.
> >>
> >> On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>>
> >>> You're right, to get those numbers on debian something is very wrong.
> >>>
> >>> Have you looked at
> >>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
> >>> What is the bottleneck on the linux machines?
> >>>
> >>> With the kind of speed you are seeing I wouldn't be surprised if it is
> >>> swapping.
> >>>
> >>> -Jonathan
> >>>
> >>> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
> >>> wrote:
> >>> > Hi,
> >>> > I wrote a few days ago and got a few good suggestions. I'm still
> >>> > seeing
> >>> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
> >>> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
> >>> > identical
> >>> > results. I've tried a mix of hardware.
> >>> > Attached are some graphs I've produced of my results which show that
> in
> >>> > OSX,
> >>> > Cassandra takes longer with a greater load but is wicked fast
> >>> > (expected).
> >>> > In the SunJDK or Open JDK on Debian I get amazingly consistent time
> >>> > taken to
> >>> > do the writes, regardless of the load and the times are always
> >>> > ridiculously
> >>> > high. It's insanely slow.
> >>> > I genuinely believe that I must be doing something very wrong in my
> >>> > Debian
> >>> > setups, but they are all vanilla installs, both 64 bit and 32 bit
> >>> > machines,
> >>> > 64bit and 32 bit installs. Cassandra packs taken from
> >>> > http://www.apache.org/dist/cassandra/debian.
> >>> > I am using Thrift, and I'm using a c# client because that's how I
> >>> > intend to
> >>> > actually use Cassandra and it seems pretty sensible.
> >>> > An example of what I'm seeing is:
> >>> > 5 Threads Each writing 100,000 Simple Entries
> >>> > OSX: 1 min 16 seconds ~ 6515 Entries / second
> >>> > Debian: 1 hour 15 seconds ~ 138 Records / second
> >>> > 15 Threads Each writing 100,000 Simple Entries
> >>> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
> >>> > Debian: 1 hour 1.5 minutes ~406 Entries / second
> >>> > 20 Threads Each Writing 100,000 Simple Entries
> >>> > OSX: 3min 19 seconds ~ 10,050 Entries / second
> >>> > Debian: 1 hour 20 seconds ~ 492 Entries / second
> >>> > If anyone has any suggestions or pointers I'd be glad to hear them.
> >>> > Thanks,
> >>> > Stu
> >>> > Attached:
> >>> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice
> format
> >>> > downloaded from Google Docs)
> >>> > 2. OSX Records per Second - a graph of how many entries get written
> per
> >>> > second for 10,000 & 100,000 entries as thread count is increased in
> >>> > OSX.
> >>> > 3. Open JDK Records per Second - the same graph but of Open JDK on
> >>> > Debian
> >>> > 4. Open JDK Total Time By Thread - the total time taken from test
> start
> >>> > to
> >>> > finish (all threads completed) to write 10,000 & 100,000 entries as
> >>> > thread
> >>> > count is increased in Debian with Open JDK
> >>> > 5. OSX Total time by Thread - same as 4, but for OSX.
> >>> >
> >>> >
> >>
> >
> >
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Heath Oderman <he...@526valley.com>.
I upgraded to 0.6 yesterday and it's bang on the same. I'll go read up on
py_stress and give it a try.
On Thu, Apr 15, 2010 at 1:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> What kind of numbers do you get from contrib/py_stress?
>
> (that's located somewhere else in 0.5, but you should really be using
> 0.6 anyway.)
>
> On Thu, Apr 15, 2010 at 12:53 PM, Heath Oderman <he...@526valley.com>
> wrote:
> > So checking it out quickly:
> > vmstat -
> > Never swaps. si and so stay at 0 during the load.
> > iostat -x
> > the %util never climbs above 0.00, but the avgrg-sz jumps bewteen samples
> > from 0 - 30 - 90 - 0 (5 second intervals)
> > top shows the cpu barely working and mem utilization is below 20%.
> > Still slow. :(
> > Thanks for the suggestions. In your article on your blog it'd be awesome
> to
> > include some implications, like "avgrg-sz over 250 may mean XXX" Even if
> > it's utterly hardware and system dependent it'd give a guy like me an
> idea
> > if what I was seeing was bad or good. :D
> > Thanks again,
> > Heath
> >
> > On Thu, Apr 15, 2010 at 1:34 PM, Heath Oderman <he...@526valley.com>
> wrote:
> >>
> >> Thanks Jonathan, I'll check this out right away.
> >>
> >> On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>>
> >>> You're right, to get those numbers on debian something is very wrong.
> >>>
> >>> Have you looked at
> >>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
> >>> What is the bottleneck on the linux machines?
> >>>
> >>> With the kind of speed you are seeing I wouldn't be surprised if it is
> >>> swapping.
> >>>
> >>> -Jonathan
> >>>
> >>> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
> >>> wrote:
> >>> > Hi,
> >>> > I wrote a few days ago and got a few good suggestions. I'm still
> >>> > seeing
> >>> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
> >>> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
> >>> > identical
> >>> > results. I've tried a mix of hardware.
> >>> > Attached are some graphs I've produced of my results which show that
> in
> >>> > OSX,
> >>> > Cassandra takes longer with a greater load but is wicked fast
> >>> > (expected).
> >>> > In the SunJDK or Open JDK on Debian I get amazingly consistent time
> >>> > taken to
> >>> > do the writes, regardless of the load and the times are always
> >>> > ridiculously
> >>> > high. It's insanely slow.
> >>> > I genuinely believe that I must be doing something very wrong in my
> >>> > Debian
> >>> > setups, but they are all vanilla installs, both 64 bit and 32 bit
> >>> > machines,
> >>> > 64bit and 32 bit installs. Cassandra packs taken from
> >>> > http://www.apache.org/dist/cassandra/debian.
> >>> > I am using Thrift, and I'm using a c# client because that's how I
> >>> > intend to
> >>> > actually use Cassandra and it seems pretty sensible.
> >>> > An example of what I'm seeing is:
> >>> > 5 Threads Each writing 100,000 Simple Entries
> >>> > OSX: 1 min 16 seconds ~ 6515 Entries / second
> >>> > Debian: 1 hour 15 seconds ~ 138 Records / second
> >>> > 15 Threads Each writing 100,000 Simple Entries
> >>> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
> >>> > Debian: 1 hour 1.5 minutes ~406 Entries / second
> >>> > 20 Threads Each Writing 100,000 Simple Entries
> >>> > OSX: 3min 19 seconds ~ 10,050 Entries / second
> >>> > Debian: 1 hour 20 seconds ~ 492 Entries / second
> >>> > If anyone has any suggestions or pointers I'd be glad to hear them.
> >>> > Thanks,
> >>> > Stu
> >>> > Attached:
> >>> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice
> format
> >>> > downloaded from Google Docs)
> >>> > 2. OSX Records per Second - a graph of how many entries get written
> per
> >>> > second for 10,000 & 100,000 entries as thread count is increased in
> >>> > OSX.
> >>> > 3. Open JDK Records per Second - the same graph but of Open JDK on
> >>> > Debian
> >>> > 4. Open JDK Total Time By Thread - the total time taken from test
> start
> >>> > to
> >>> > finish (all threads completed) to write 10,000 & 100,000 entries as
> >>> > thread
> >>> > count is increased in Debian with Open JDK
> >>> > 5. OSX Total time by Thread - same as 4, but for OSX.
> >>> >
> >>> >
> >>
> >
> >
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Jonathan Ellis <jb...@gmail.com>.
What kind of numbers do you get from contrib/py_stress?
(that's located somewhere else in 0.5, but you should really be using
0.6 anyway.)
On Thu, Apr 15, 2010 at 12:53 PM, Heath Oderman <he...@526valley.com> wrote:
> So checking it out quickly:
> vmstat -
> Never swaps. si and so stay at 0 during the load.
> iostat -x
> the %util never climbs above 0.00, but the avgrg-sz jumps bewteen samples
> from 0 - 30 - 90 - 0 (5 second intervals)
> top shows the cpu barely working and mem utilization is below 20%.
> Still slow. :(
> Thanks for the suggestions. In your article on your blog it'd be awesome to
> include some implications, like "avgrg-sz over 250 may mean XXX" Even if
> it's utterly hardware and system dependent it'd give a guy like me an idea
> if what I was seeing was bad or good. :D
> Thanks again,
> Heath
>
> On Thu, Apr 15, 2010 at 1:34 PM, Heath Oderman <he...@526valley.com> wrote:
>>
>> Thanks Jonathan, I'll check this out right away.
>>
>> On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>> You're right, to get those numbers on debian something is very wrong.
>>>
>>> Have you looked at
>>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
>>> What is the bottleneck on the linux machines?
>>>
>>> With the kind of speed you are seeing I wouldn't be surprised if it is
>>> swapping.
>>>
>>> -Jonathan
>>>
>>> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
>>> wrote:
>>> > Hi,
>>> > I wrote a few days ago and got a few good suggestions. I'm still
>>> > seeing
>>> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
>>> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
>>> > identical
>>> > results. I've tried a mix of hardware.
>>> > Attached are some graphs I've produced of my results which show that in
>>> > OSX,
>>> > Cassandra takes longer with a greater load but is wicked fast
>>> > (expected).
>>> > In the SunJDK or Open JDK on Debian I get amazingly consistent time
>>> > taken to
>>> > do the writes, regardless of the load and the times are always
>>> > ridiculously
>>> > high. It's insanely slow.
>>> > I genuinely believe that I must be doing something very wrong in my
>>> > Debian
>>> > setups, but they are all vanilla installs, both 64 bit and 32 bit
>>> > machines,
>>> > 64bit and 32 bit installs. Cassandra packs taken from
>>> > http://www.apache.org/dist/cassandra/debian.
>>> > I am using Thrift, and I'm using a c# client because that's how I
>>> > intend to
>>> > actually use Cassandra and it seems pretty sensible.
>>> > An example of what I'm seeing is:
>>> > 5 Threads Each writing 100,000 Simple Entries
>>> > OSX: 1 min 16 seconds ~ 6515 Entries / second
>>> > Debian: 1 hour 15 seconds ~ 138 Records / second
>>> > 15 Threads Each writing 100,000 Simple Entries
>>> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
>>> > Debian: 1 hour 1.5 minutes ~406 Entries / second
>>> > 20 Threads Each Writing 100,000 Simple Entries
>>> > OSX: 3min 19 seconds ~ 10,050 Entries / second
>>> > Debian: 1 hour 20 seconds ~ 492 Entries / second
>>> > If anyone has any suggestions or pointers I'd be glad to hear them.
>>> > Thanks,
>>> > Stu
>>> > Attached:
>>> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
>>> > downloaded from Google Docs)
>>> > 2. OSX Records per Second - a graph of how many entries get written per
>>> > second for 10,000 & 100,000 entries as thread count is increased in
>>> > OSX.
>>> > 3. Open JDK Records per Second - the same graph but of Open JDK on
>>> > Debian
>>> > 4. Open JDK Total Time By Thread - the total time taken from test start
>>> > to
>>> > finish (all threads completed) to write 10,000 & 100,000 entries as
>>> > thread
>>> > count is increased in Debian with Open JDK
>>> > 5. OSX Total time by Thread - same as 4, but for OSX.
>>> >
>>> >
>>
>
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Heath Oderman <he...@526valley.com>.
So checking it out quickly:
vmstat -
Never swaps. si and so stay at 0 during the load.
iostat -x
the %util never climbs above 0.00, but the avgrg-sz jumps bewteen samples
from 0 - 30 - 90 - 0 (5 second intervals)
top shows the cpu barely working and mem utilization is below 20%.
Still slow. :(
Thanks for the suggestions. In your article on your blog it'd be awesome to
include some implications, like "avgrg-sz over 250 may mean XXX" Even if
it's utterly hardware and system dependent it'd give a guy like me an idea
if what I was seeing was bad or good. :D
Thanks again,
Heath
On Thu, Apr 15, 2010 at 1:34 PM, Heath Oderman <he...@526valley.com> wrote:
> Thanks Jonathan, I'll check this out right away.
>
>
> On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> You're right, to get those numbers on debian something is very wrong.
>>
>> Have you looked at
>> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
>> What is the bottleneck on the linux machines?
>>
>> With the kind of speed you are seeing I wouldn't be surprised if it is
>> swapping.
>>
>> -Jonathan
>>
>> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
>> wrote:
>> > Hi,
>> > I wrote a few days ago and got a few good suggestions. I'm still seeing
>> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
>> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
>> identical
>> > results. I've tried a mix of hardware.
>> > Attached are some graphs I've produced of my results which show that in
>> OSX,
>> > Cassandra takes longer with a greater load but is wicked fast
>> (expected).
>> > In the SunJDK or Open JDK on Debian I get amazingly consistent time
>> taken to
>> > do the writes, regardless of the load and the times are always
>> ridiculously
>> > high. It's insanely slow.
>> > I genuinely believe that I must be doing something very wrong in my
>> Debian
>> > setups, but they are all vanilla installs, both 64 bit and 32 bit
>> machines,
>> > 64bit and 32 bit installs. Cassandra packs taken from
>> > http://www.apache.org/dist/cassandra/debian.
>> > I am using Thrift, and I'm using a c# client because that's how I intend
>> to
>> > actually use Cassandra and it seems pretty sensible.
>> > An example of what I'm seeing is:
>> > 5 Threads Each writing 100,000 Simple Entries
>> > OSX: 1 min 16 seconds ~ 6515 Entries / second
>> > Debian: 1 hour 15 seconds ~ 138 Records / second
>> > 15 Threads Each writing 100,000 Simple Entries
>> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
>> > Debian: 1 hour 1.5 minutes ~406 Entries / second
>> > 20 Threads Each Writing 100,000 Simple Entries
>> > OSX: 3min 19 seconds ~ 10,050 Entries / second
>> > Debian: 1 hour 20 seconds ~ 492 Entries / second
>> > If anyone has any suggestions or pointers I'd be glad to hear them.
>> > Thanks,
>> > Stu
>> > Attached:
>> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
>> > downloaded from Google Docs)
>> > 2. OSX Records per Second - a graph of how many entries get written per
>> > second for 10,000 & 100,000 entries as thread count is increased in OSX.
>> > 3. Open JDK Records per Second - the same graph but of Open JDK on
>> Debian
>> > 4. Open JDK Total Time By Thread - the total time taken from test start
>> to
>> > finish (all threads completed) to write 10,000 & 100,000 entries as
>> thread
>> > count is increased in Debian with Open JDK
>> > 5. OSX Total time by Thread - same as 4, but for OSX.
>> >
>> >
>>
>
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Heath Oderman <he...@526valley.com>.
Thanks Jonathan, I'll check this out right away.
On Thu, Apr 15, 2010 at 1:32 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> You're right, to get those numbers on debian something is very wrong.
>
> Have you looked at
> http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
> What is the bottleneck on the linux machines?
>
> With the kind of speed you are seeing I wouldn't be surprised if it is
> swapping.
>
> -Jonathan
>
> On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com>
> wrote:
> > Hi,
> > I wrote a few days ago and got a few good suggestions. I'm still seeing
> > dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
> > I've tried on Debian with the Sun JRE and the Open JDK with nearly
> identical
> > results. I've tried a mix of hardware.
> > Attached are some graphs I've produced of my results which show that in
> OSX,
> > Cassandra takes longer with a greater load but is wicked fast (expected).
> > In the SunJDK or Open JDK on Debian I get amazingly consistent time taken
> to
> > do the writes, regardless of the load and the times are always
> ridiculously
> > high. It's insanely slow.
> > I genuinely believe that I must be doing something very wrong in my
> Debian
> > setups, but they are all vanilla installs, both 64 bit and 32 bit
> machines,
> > 64bit and 32 bit installs. Cassandra packs taken from
> > http://www.apache.org/dist/cassandra/debian.
> > I am using Thrift, and I'm using a c# client because that's how I intend
> to
> > actually use Cassandra and it seems pretty sensible.
> > An example of what I'm seeing is:
> > 5 Threads Each writing 100,000 Simple Entries
> > OSX: 1 min 16 seconds ~ 6515 Entries / second
> > Debian: 1 hour 15 seconds ~ 138 Records / second
> > 15 Threads Each writing 100,000 Simple Entries
> > OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
> > Debian: 1 hour 1.5 minutes ~406 Entries / second
> > 20 Threads Each Writing 100,000 Simple Entries
> > OSX: 3min 19 seconds ~ 10,050 Entries / second
> > Debian: 1 hour 20 seconds ~ 492 Entries / second
> > If anyone has any suggestions or pointers I'd be glad to hear them.
> > Thanks,
> > Stu
> > Attached:
> > 1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
> > downloaded from Google Docs)
> > 2. OSX Records per Second - a graph of how many entries get written per
> > second for 10,000 & 100,000 entries as thread count is increased in OSX.
> > 3. Open JDK Records per Second - the same graph but of Open JDK on Debian
> > 4. Open JDK Total Time By Thread - the total time taken from test start
> to
> > finish (all threads completed) to write 10,000 & 100,000 entries as
> thread
> > count is increased in Debian with Open JDK
> > 5. OSX Total time by Thread - same as 4, but for OSX.
> >
> >
>
Re: New User: OSX vs. Debian on Cassandra 0.5.0 with Thrift
Posted by Jonathan Ellis <jb...@gmail.com>.
You're right, to get those numbers on debian something is very wrong.
Have you looked at
http://spyced.blogspot.com/2010/01/linux-performance-basics.html ?
What is the bottleneck on the linux machines?
With the kind of speed you are seeing I wouldn't be surprised if it is swapping.
-Jonathan
On Tue, Apr 13, 2010 at 11:38 PM, Heath Oderman <he...@526valley.com> wrote:
> Hi,
> I wrote a few days ago and got a few good suggestions. I'm still seeing
> dramatic differences between Cassandra 0.5.0 on OSX vs. Debian Linux.
> I've tried on Debian with the Sun JRE and the Open JDK with nearly identical
> results. I've tried a mix of hardware.
> Attached are some graphs I've produced of my results which show that in OSX,
> Cassandra takes longer with a greater load but is wicked fast (expected).
> In the SunJDK or Open JDK on Debian I get amazingly consistent time taken to
> do the writes, regardless of the load and the times are always ridiculously
> high. It's insanely slow.
> I genuinely believe that I must be doing something very wrong in my Debian
> setups, but they are all vanilla installs, both 64 bit and 32 bit machines,
> 64bit and 32 bit installs. Cassandra packs taken from
> http://www.apache.org/dist/cassandra/debian.
> I am using Thrift, and I'm using a c# client because that's how I intend to
> actually use Cassandra and it seems pretty sensible.
> An example of what I'm seeing is:
> 5 Threads Each writing 100,000 Simple Entries
> OSX: 1 min 16 seconds ~ 6515 Entries / second
> Debian: 1 hour 15 seconds ~ 138 Records / second
> 15 Threads Each writing 100,000 Simple Entries
> OSX: 2min 30 seconds seconds writing ~10,000 Entries / second
> Debian: 1 hour 1.5 minutes ~406 Entries / second
> 20 Threads Each Writing 100,000 Simple Entries
> OSX: 3min 19 seconds ~ 10,050 Entries / second
> Debian: 1 hour 20 seconds ~ 492 Entries / second
> If anyone has any suggestions or pointers I'd be glad to hear them.
> Thanks,
> Stu
> Attached:
> 1. CassLoadTesting.ods (all my results and graphs in OpenOffice format
> downloaded from Google Docs)
> 2. OSX Records per Second - a graph of how many entries get written per
> second for 10,000 & 100,000 entries as thread count is increased in OSX.
> 3. Open JDK Records per Second - the same graph but of Open JDK on Debian
> 4. Open JDK Total Time By Thread - the total time taken from test start to
> finish (all threads completed) to write 10,000 & 100,000 entries as thread
> count is increased in Debian with Open JDK
> 5. OSX Total time by Thread - same as 4, but for OSX.
>
>