You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jay Kreps <ja...@gmail.com> on 2011/10/24 21:06:51 UTC
perf testing
Neha and I have been doing so work on perf testing. My past experience with
these perf and integration testing frameworks is that
1. This kind of testing is extremely important. At least as important as
unit testing. A lot of the bugs that are caught in production could be
caught be good integration tests but likely will never be caught be unit
tests.
2. It is hard to get all your pieces scripted up so that you can fully
automate the perf analysis you want to do and run this every day.
3. It requires dedicated hardware that doesn't change from day to day.
4. One of the biggest problems is that perf code is always kind of an
afterthought. As a result one never gets to a framework that is good enough
to use. Instead you keep re-writing the same test harnesses over and over
but with little tweaks for each new test you need to run, then throwing that
code away because it is so specific it can't be reused.
To hopefully help I started a wiki where we could work out some ideas for
this. The idea is basically just to dump out all the stats we have now to
CSV and do some analysis in R. Then script this up in a way that we can
craft "test scenarios" and run a bunch of these different configurations.
Wiki here:
https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
Would love to hear people's thoughts.
-Jay
Re: perf testing
Posted by Chris Burroughs <ch...@gmail.com>.
On 10/24/2011 03:27 PM, Neha Narkhede wrote:
> % busy - indicates how busy the disks are.
To my knowledge there isn't a god way to get this if disks is > 1.
iostat's %util becomes "percent of time waiting on at least one disk".
Re: perf testing
Posted by Jay Kreps <ja...@gmail.com>.
Yeah that is a good point can you update the wiki. Sounds like we need
iostat.
-Jay
On Mon, Oct 24, 2011 at 12:27 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Thanks for starting this discussion Jay !
>
> >> dstat - This is the existing perf tool to get read/write/fs stats
>
> My only concern about dstat is that it doesn't seem to give some
> important IO stats like
>
> avgrq-sz - size of the write request after merging several writes
> together into one big write ( indicates how sequential the IO is)
> wrqm_s - write requests merged per sec (also, could indicate how much
> of the total IO is sequential and can be grouped into one write on the
> disk)
> w/s - write IO requests per sec (indicates how many flushes the kafka
> broker is doing per sec)
> % busy - indicates how busy the disks are.
>
> At the very least, %busy is what we would like to know. Maybe I
> couldn't pass the right options to dstat.
>
> Does anyone have more ideas about dstat or some other perf tool,
> preferably one that would work on Linux ?
>
> Thanks,
> Neha
>
> On Mon, Oct 24, 2011 at 12:06 PM, Jay Kreps <ja...@gmail.com> wrote:
> > Neha and I have been doing so work on perf testing. My past experience
> with
> > these perf and integration testing frameworks is that
> >
> > 1. This kind of testing is extremely important. At least as important
> as
> > unit testing. A lot of the bugs that are caught in production could be
> > caught be good integration tests but likely will never be caught be
> unit
> > tests.
> > 2. It is hard to get all your pieces scripted up so that you can fully
> > automate the perf analysis you want to do and run this every day.
> > 3. It requires dedicated hardware that doesn't change from day to day.
> > 4. One of the biggest problems is that perf code is always kind of an
> > afterthought. As a result one never gets to a framework that is good
> enough
> > to use. Instead you keep re-writing the same test harnesses over and
> over
> > but with little tweaks for each new test you need to run, then throwing
> that
> > code away because it is so specific it can't be reused.
> >
> > To hopefully help I started a wiki where we could work out some ideas for
> > this. The idea is basically just to dump out all the stats we have now to
> > CSV and do some analysis in R. Then script this up in a way that we can
> > craft "test scenarios" and run a bunch of these different configurations.
> >
> > Wiki here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
> >
> > Would love to hear people's thoughts.
> >
> > -Jay
> >
>
Re: perf testing
Posted by Chris Burroughs <ch...@gmail.com>.
On 10/24/2011 03:27 PM, Neha Narkhede wrote:
> % busy - indicates how busy the disks are.
To my knowledge there isn't a god way to get this if disks is > 1.
iostat's %util becomes "percent of time waiting on at least one disk".
Re: perf testing
Posted by Neha Narkhede <ne...@gmail.com>.
Thanks for starting this discussion Jay !
>> dstat - This is the existing perf tool to get read/write/fs stats
My only concern about dstat is that it doesn't seem to give some
important IO stats like
avgrq-sz - size of the write request after merging several writes
together into one big write ( indicates how sequential the IO is)
wrqm_s - write requests merged per sec (also, could indicate how much
of the total IO is sequential and can be grouped into one write on the
disk)
w/s - write IO requests per sec (indicates how many flushes the kafka
broker is doing per sec)
% busy - indicates how busy the disks are.
At the very least, %busy is what we would like to know. Maybe I
couldn't pass the right options to dstat.
Does anyone have more ideas about dstat or some other perf tool,
preferably one that would work on Linux ?
Thanks,
Neha
On Mon, Oct 24, 2011 at 12:06 PM, Jay Kreps <ja...@gmail.com> wrote:
> Neha and I have been doing so work on perf testing. My past experience with
> these perf and integration testing frameworks is that
>
> 1. This kind of testing is extremely important. At least as important as
> unit testing. A lot of the bugs that are caught in production could be
> caught be good integration tests but likely will never be caught be unit
> tests.
> 2. It is hard to get all your pieces scripted up so that you can fully
> automate the perf analysis you want to do and run this every day.
> 3. It requires dedicated hardware that doesn't change from day to day.
> 4. One of the biggest problems is that perf code is always kind of an
> afterthought. As a result one never gets to a framework that is good enough
> to use. Instead you keep re-writing the same test harnesses over and over
> but with little tweaks for each new test you need to run, then throwing that
> code away because it is so specific it can't be reused.
>
> To hopefully help I started a wiki where we could work out some ideas for
> this. The idea is basically just to dump out all the stats we have now to
> CSV and do some analysis in R. Then script this up in a way that we can
> craft "test scenarios" and run a bunch of these different configurations.
>
> Wiki here:
> https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
>
> Would love to hear people's thoughts.
>
> -Jay
>
Re: perf testing
Posted by Jeffrey Damick <je...@gmail.com>.
In the past I've used jstat (-gc, etc..) & jmap, but using VisualVM with
jstatd is a little friendlier to consume if you use the visual gc plugin..
On Thu, Oct 27, 2011 at 10:55 AM, Jay Kreps <ja...@gmail.com> wrote:
> That is a good point, what stats would you recommend. Ideally it would be
> good to get something we can pull from jmx.
>
> -jay
>
> On Thu, Oct 27, 2011 at 7:09 AM, Jeffrey Damick <jeffreydamick@gmail.com
> >wrote:
>
> > One thought, I don't see any points related to jvm heap / gc profiling,
> it
> > may not be a major concern at this point but it probably makes sense to
> > have
> > a baseline..
> >
> >
> > On Mon, Oct 24, 2011 at 3:06 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > Neha and I have been doing so work on perf testing. My past experience
> > with
> > > these perf and integration testing frameworks is that
> > >
> > > 1. This kind of testing is extremely important. At least as important
> > as
> > > unit testing. A lot of the bugs that are caught in production could
> be
> > > caught be good integration tests but likely will never be caught be
> > unit
> > > tests.
> > > 2. It is hard to get all your pieces scripted up so that you can
> fully
> > > automate the perf analysis you want to do and run this every day.
> > > 3. It requires dedicated hardware that doesn't change from day to
> day.
> > > 4. One of the biggest problems is that perf code is always kind of an
> > > afterthought. As a result one never gets to a framework that is good
> > > enough
> > > to use. Instead you keep re-writing the same test harnesses over and
> > over
> > > but with little tweaks for each new test you need to run, then
> throwing
> > > that
> > > code away because it is so specific it can't be reused.
> > >
> > > To hopefully help I started a wiki where we could work out some ideas
> for
> > > this. The idea is basically just to dump out all the stats we have now
> to
> > > CSV and do some analysis in R. Then script this up in a way that we can
> > > craft "test scenarios" and run a bunch of these different
> configurations.
> > >
> > > Wiki here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
> > >
> > > Would love to hear people's thoughts.
> > >
> > > -Jay
> > >
> >
>
Re: perf testing
Posted by Chris Burroughs <ch...@gmail.com>.
On 10/27/2011 10:55 AM, Jay Kreps wrote:
> That is a good point, what stats would you recommend. Ideally it would be
> good to get something we can pull from jmx.
Not from JMX, but other than dealing with the less than friendly format
format I've had luck in the past deriving meaning from the various
"print gc info" settings:
http://svn.apache.org/viewvc/cassandra/trunk/conf/cassandra-env.sh?r1=1043261&r2=1043262&
I agree that heap turning is great to get to, but I think latency
measurements will already be able to identify if we are experience
problems with pauses.
Re: perf testing
Posted by Jay Kreps <ja...@gmail.com>.
That is a good point, what stats would you recommend. Ideally it would be
good to get something we can pull from jmx.
-jay
On Thu, Oct 27, 2011 at 7:09 AM, Jeffrey Damick <je...@gmail.com>wrote:
> One thought, I don't see any points related to jvm heap / gc profiling, it
> may not be a major concern at this point but it probably makes sense to
> have
> a baseline..
>
>
> On Mon, Oct 24, 2011 at 3:06 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > Neha and I have been doing so work on perf testing. My past experience
> with
> > these perf and integration testing frameworks is that
> >
> > 1. This kind of testing is extremely important. At least as important
> as
> > unit testing. A lot of the bugs that are caught in production could be
> > caught be good integration tests but likely will never be caught be
> unit
> > tests.
> > 2. It is hard to get all your pieces scripted up so that you can fully
> > automate the perf analysis you want to do and run this every day.
> > 3. It requires dedicated hardware that doesn't change from day to day.
> > 4. One of the biggest problems is that perf code is always kind of an
> > afterthought. As a result one never gets to a framework that is good
> > enough
> > to use. Instead you keep re-writing the same test harnesses over and
> over
> > but with little tweaks for each new test you need to run, then throwing
> > that
> > code away because it is so specific it can't be reused.
> >
> > To hopefully help I started a wiki where we could work out some ideas for
> > this. The idea is basically just to dump out all the stats we have now to
> > CSV and do some analysis in R. Then script this up in a way that we can
> > craft "test scenarios" and run a bunch of these different configurations.
> >
> > Wiki here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
> >
> > Would love to hear people's thoughts.
> >
> > -Jay
> >
>
Re: perf testing
Posted by Jeffrey Damick <je...@gmail.com>.
One thought, I don't see any points related to jvm heap / gc profiling, it
may not be a major concern at this point but it probably makes sense to have
a baseline..
On Mon, Oct 24, 2011 at 3:06 PM, Jay Kreps <ja...@gmail.com> wrote:
> Neha and I have been doing so work on perf testing. My past experience with
> these perf and integration testing frameworks is that
>
> 1. This kind of testing is extremely important. At least as important as
> unit testing. A lot of the bugs that are caught in production could be
> caught be good integration tests but likely will never be caught be unit
> tests.
> 2. It is hard to get all your pieces scripted up so that you can fully
> automate the perf analysis you want to do and run this every day.
> 3. It requires dedicated hardware that doesn't change from day to day.
> 4. One of the biggest problems is that perf code is always kind of an
> afterthought. As a result one never gets to a framework that is good
> enough
> to use. Instead you keep re-writing the same test harnesses over and over
> but with little tweaks for each new test you need to run, then throwing
> that
> code away because it is so specific it can't be reused.
>
> To hopefully help I started a wiki where we could work out some ideas for
> this. The idea is basically just to dump out all the stats we have now to
> CSV and do some analysis in R. Then script this up in a way that we can
> craft "test scenarios" and run a bunch of these different configurations.
>
> Wiki here:
> https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
>
> Would love to hear people's thoughts.
>
> -Jay
>
Re: perf testing
Posted by Neha Narkhede <ne...@gmail.com>.
Thanks for starting this discussion Jay !
>> dstat - This is the existing perf tool to get read/write/fs stats
My only concern about dstat is that it doesn't seem to give some
important IO stats like
avgrq-sz - size of the write request after merging several writes
together into one big write ( indicates how sequential the IO is)
wrqm_s - write requests merged per sec (also, could indicate how much
of the total IO is sequential and can be grouped into one write on the
disk)
w/s - write IO requests per sec (indicates how many flushes the kafka
broker is doing per sec)
% busy - indicates how busy the disks are.
At the very least, %busy is what we would like to know. Maybe I
couldn't pass the right options to dstat.
Does anyone have more ideas about dstat or some other perf tool,
preferably one that would work on Linux ?
Thanks,
Neha
On Mon, Oct 24, 2011 at 12:06 PM, Jay Kreps <ja...@gmail.com> wrote:
> Neha and I have been doing so work on perf testing. My past experience with
> these perf and integration testing frameworks is that
>
> 1. This kind of testing is extremely important. At least as important as
> unit testing. A lot of the bugs that are caught in production could be
> caught be good integration tests but likely will never be caught be unit
> tests.
> 2. It is hard to get all your pieces scripted up so that you can fully
> automate the perf analysis you want to do and run this every day.
> 3. It requires dedicated hardware that doesn't change from day to day.
> 4. One of the biggest problems is that perf code is always kind of an
> afterthought. As a result one never gets to a framework that is good enough
> to use. Instead you keep re-writing the same test harnesses over and over
> but with little tweaks for each new test you need to run, then throwing that
> code away because it is so specific it can't be reused.
>
> To hopefully help I started a wiki where we could work out some ideas for
> this. The idea is basically just to dump out all the stats we have now to
> CSV and do some analysis in R. Then script this up in a way that we can
> craft "test scenarios" and run a bunch of these different configurations.
>
> Wiki here:
> https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing
>
> Would love to hear people's thoughts.
>
> -Jay
>