slow producing on ec2

[Pierre-Yves Ritschard <py...@spootnik.org>]

Hi,

One of my cluster runs on ec2 and I noticed that I have rather slow
throughput, less than 500 per second. I isolated the trouble on the
following setup

1 producer instance (m1.large)
5 consumer instances also running a broker (m2.xlarge)
1 zookeeper instance (m1.large)

I am using a sync producer and have 60 partitions.

On my local box, with everything running I would several thousand per second.

Re: slow producing on ec2

[Felix GV <fe...@mate1inc.com>]

One order of magnitude of difference sounds about right, though, no?

I don't know. I'm by no means a Kafka expert and we don't know much about
the other variables of your situation, so it's hard to tell (at least for
me) if there could be another explanation...

--
Felix



On Tue, Jan 10, 2012 at 5:30 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:

> Surely,
>
> I just didn't expect such dramatically low numbers
>
> On Tue, Jan 10, 2012 at 11:27 PM, Felix GV <fe...@mate1inc.com> wrote:
> > Maybe I'm overlooking something, but the first thing that came to my mind
> > is: wouldn't you get no network latency at all on your local box if
> > everything runs on the same machine?
> >
> > On EC2, the network latency would bring your overal throughput down,
> > especially with a sync producer, wouldn't it?
> >
> > --
> > Felix
> >
> >
> >
> > On Tue, Jan 10, 2012 at 5:19 PM, Pierre-Yves Ritschard <pyr@spootnik.org
> >wrote:
> >
> >> Hi,
> >>
> >> One of my cluster runs on ec2 and I noticed that I have rather slow
> >> throughput, less than 500 per second. I isolated the trouble on the
> >> following setup
> >>
> >> 1 producer instance (m1.large)
> >> 5 consumer instances also running a broker (m2.xlarge)
> >> 1 zookeeper instance (m1.large)
> >>
> >> I am using a sync producer and have 60 partitions.
> >>
> >> On my local box, with everything running I would several thousand per
> >> second.
> >>
>

Re: slow producing on ec2

[Pierre-Yves Ritschard <py...@spootnik.org>]

Surely,

I just didn't expect such dramatically low numbers

On Tue, Jan 10, 2012 at 11:27 PM, Felix GV <fe...@mate1inc.com> wrote:
> Maybe I'm overlooking something, but the first thing that came to my mind
> is: wouldn't you get no network latency at all on your local box if
> everything runs on the same machine?
>
> On EC2, the network latency would bring your overal throughput down,
> especially with a sync producer, wouldn't it?
>
> --
> Felix
>
>
>
> On Tue, Jan 10, 2012 at 5:19 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:
>
>> Hi,
>>
>> One of my cluster runs on ec2 and I noticed that I have rather slow
>> throughput, less than 500 per second. I isolated the trouble on the
>> following setup
>>
>> 1 producer instance (m1.large)
>> 5 consumer instances also running a broker (m2.xlarge)
>> 1 zookeeper instance (m1.large)
>>
>> I am using a sync producer and have 60 partitions.
>>
>> On my local box, with everything running I would several thousand per
>> second.
>>

Re: slow producing on ec2

[Felix GV <fe...@mate1inc.com>]

Maybe I'm overlooking something, but the first thing that came to my mind
is: wouldn't you get no network latency at all on your local box if
everything runs on the same machine?

On EC2, the network latency would bring your overal throughput down,
especially with a sync producer, wouldn't it?

--
Felix



On Tue, Jan 10, 2012 at 5:19 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:

> Hi,
>
> One of my cluster runs on ec2 and I noticed that I have rather slow
> throughput, less than 500 per second. I isolated the trouble on the
> following setup
>
> 1 producer instance (m1.large)
> 5 consumer instances also running a broker (m2.xlarge)
> 1 zookeeper instance (m1.large)
>
> I am using a sync producer and have 60 partitions.
>
> On my local box, with everything running I would several thousand per
> second.
>

Re: slow producing on ec2

[Jay Kreps <ja...@gmail.com>]

The majority of the problems we have seen have been due to i/o bottlenecks
on the server. iostat will uncover these, though if you aren't familiar
with it you may need to read through some docs. If iostat show high mb/sec,
high number of operations (reads or writes) per second or high service time
that is likely the issue.

-Jay

On Tue, Jan 10, 2012 at 2:55 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:

> will try this, thanks
>
> On Tue, Jan 10, 2012 at 11:52 PM, Jun Rao <ju...@gmail.com> wrote:
> > Pierre-Yves,
> >
> > I assume that this is a producer problem. If so, first, make sure flush
> > interval is not too small. Second, you can enable debug logging in
> > FileMessageSet to see log flush time. Then you can see if it's higher
> than
> > what you expect.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 10, 2012 at 2:19 PM, Pierre-Yves Ritschard <pyr@spootnik.org
> >wrote:
> >
> >> Hi,
> >>
> >> One of my cluster runs on ec2 and I noticed that I have rather slow
> >> throughput, less than 500 per second. I isolated the trouble on the
> >> following setup
> >>
> >> 1 producer instance (m1.large)
> >> 5 consumer instances also running a broker (m2.xlarge)
> >> 1 zookeeper instance (m1.large)
> >>
> >> I am using a sync producer and have 60 partitions.
> >>
> >> On my local box, with everything running I would several thousand per
> >> second.
> >>
>

Re: slow producing on ec2

[Pierre-Yves Ritschard <py...@spootnik.org>]

will try this, thanks

On Tue, Jan 10, 2012 at 11:52 PM, Jun Rao <ju...@gmail.com> wrote:
> Pierre-Yves,
>
> I assume that this is a producer problem. If so, first, make sure flush
> interval is not too small. Second, you can enable debug logging in
> FileMessageSet to see log flush time. Then you can see if it's higher than
> what you expect.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 10, 2012 at 2:19 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:
>
>> Hi,
>>
>> One of my cluster runs on ec2 and I noticed that I have rather slow
>> throughput, less than 500 per second. I isolated the trouble on the
>> following setup
>>
>> 1 producer instance (m1.large)
>> 5 consumer instances also running a broker (m2.xlarge)
>> 1 zookeeper instance (m1.large)
>>
>> I am using a sync producer and have 60 partitions.
>>
>> On my local box, with everything running I would several thousand per
>> second.
>>

Re: slow producing on ec2

[Jun Rao <ju...@gmail.com>]

Pierre-Yves,

I assume that this is a producer problem. If so, first, make sure flush
interval is not too small. Second, you can enable debug logging in
FileMessageSet to see log flush time. Then you can see if it's higher than
what you expect.

Thanks,

Jun


On Tue, Jan 10, 2012 at 2:19 PM, Pierre-Yves Ritschard <py...@spootnik.org>wrote:

> Hi,
>
> One of my cluster runs on ec2 and I noticed that I have rather slow
> throughput, less than 500 per second. I isolated the trouble on the
> following setup
>
> 1 producer instance (m1.large)
> 5 consumer instances also running a broker (m2.xlarge)
> 1 zookeeper instance (m1.large)
>
> I am using a sync producer and have 60 partitions.
>
> On my local box, with everything running I would several thousand per
> second.
>