One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Philip O'Toole <ph...@loggly.com>]

Hello -- we're using 0.72. We're looking at the source, but want to be sure. :-)

We create a single ConsumerConnector, call createMessageStreams, and
hand the streams off to individual threads. If one of those threads
calls next() on a stream, gets some messages, and then *blocks* in
some subsequent operation (and blocks for minutes), can it potentially
cause all other threads (calling next() on other streams) to block
too? Does something inside the ConsumerConnector block all other
stream processing? This would explain some behaviour we're seeing.

Thanks,

Philip

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Philip O'Toole <ph...@loggly.com>]

Also, what is it in the ConsumerConnection that causes this behaviour?

I'm proposing here that we move to a model of ConsumerConnection per
thread. This will decouple the flow for each partition, and allow each
to flow, right? We only have one topic on the cluster.

Thanks,

Philip

On Wed, Jun 12, 2013 at 9:10 PM, Philip O'Toole <ph...@loggly.com> wrote:
> Jun -- thanks.
>
> But if the topic is the same, doesn't each thread get a partition?
> Isn't that how it works?
>
> Philip
>
> On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao <ju...@gmail.com> wrote:
>> Yes, when the consumer is consuming multiple topics, if one thread stops
>> consuming topic 1, it can prevent new data getting into the consumer for
>> topic 2.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <ph...@loggly.com> wrote:
>>
>>> Hello -- we're using 0.72. We're looking at the source, but want to be
>>> sure. :-)
>>>
>>> We create a single ConsumerConnector, call createMessageStreams, and
>>> hand the streams off to individual threads. If one of those threads
>>> calls next() on a stream, gets some messages, and then *blocks* in
>>> some subsequent operation (and blocks for minutes), can it potentially
>>> cause all other threads (calling next() on other streams) to block
>>> too? Does something inside the ConsumerConnector block all other
>>> stream processing? This would explain some behaviour we're seeing.
>>>
>>> Thanks,
>>>
>>> Philip
>>>

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Philip O'Toole <ph...@loggly.com>]

Jun - thanks again. This is very helpful. 

Philip

On Jun 12, 2013, at 9:50 PM, Jun Rao <ju...@gmail.com> wrote:

> Actually, you are right. This can happen on a single topic too, if you have
> more than one consumer thread. Each consumer thread pulls data from a
> blocking queue, one or more fetchers are putting data into the queue. Say,
> you have two consumer threads and two partitions from the same broker.
> There is a single fetcher that fetches both partitions and it will put one
> partition's data into a separate queue. So, if one thread stops consuming
> data, it's queue will be full at some point. This will block the fetcher
> from putting the data into the other queue.
> 
> Thanks,
> 
> Jun
> 
> 
> On Wed, Jun 12, 2013 at 9:10 PM, Philip O'Toole <ph...@loggly.com> wrote:
> 
>> Jun -- thanks.
>> 
>> But if the topic is the same, doesn't each thread get a partition?
>> Isn't that how it works?
>> 
>> Philip
>> 
>> On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao <ju...@gmail.com> wrote:
>>> Yes, when the consumer is consuming multiple topics, if one thread stops
>>> consuming topic 1, it can prevent new data getting into the consumer for
>>> topic 2.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> 
>>> On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <ph...@loggly.com>
>> wrote:
>>> 
>>>> Hello -- we're using 0.72. We're looking at the source, but want to be
>>>> sure. :-)
>>>> 
>>>> We create a single ConsumerConnector, call createMessageStreams, and
>>>> hand the streams off to individual threads. If one of those threads
>>>> calls next() on a stream, gets some messages, and then *blocks* in
>>>> some subsequent operation (and blocks for minutes), can it potentially
>>>> cause all other threads (calling next() on other streams) to block
>>>> too? Does something inside the ConsumerConnector block all other
>>>> stream processing? This would explain some behaviour we're seeing.
>>>> 
>>>> Thanks,
>>>> 
>>>> Philip
>> 

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Jun Rao <ju...@gmail.com>]

Actually, you are right. This can happen on a single topic too, if you have
more than one consumer thread. Each consumer thread pulls data from a
blocking queue, one or more fetchers are putting data into the queue. Say,
you have two consumer threads and two partitions from the same broker.
There is a single fetcher that fetches both partitions and it will put one
partition's data into a separate queue. So, if one thread stops consuming
data, it's queue will be full at some point. This will block the fetcher
from putting the data into the other queue.

Thanks,

Jun


On Wed, Jun 12, 2013 at 9:10 PM, Philip O'Toole <ph...@loggly.com> wrote:

> Jun -- thanks.
>
> But if the topic is the same, doesn't each thread get a partition?
> Isn't that how it works?
>
> Philip
>
> On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao <ju...@gmail.com> wrote:
> > Yes, when the consumer is consuming multiple topics, if one thread stops
> > consuming topic 1, it can prevent new data getting into the consumer for
> > topic 2.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <ph...@loggly.com>
> wrote:
> >
> >> Hello -- we're using 0.72. We're looking at the source, but want to be
> >> sure. :-)
> >>
> >> We create a single ConsumerConnector, call createMessageStreams, and
> >> hand the streams off to individual threads. If one of those threads
> >> calls next() on a stream, gets some messages, and then *blocks* in
> >> some subsequent operation (and blocks for minutes), can it potentially
> >> cause all other threads (calling next() on other streams) to block
> >> too? Does something inside the ConsumerConnector block all other
> >> stream processing? This would explain some behaviour we're seeing.
> >>
> >> Thanks,
> >>
> >> Philip
> >>
>

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Philip O'Toole <ph...@loggly.com>]

Jun -- thanks.

But if the topic is the same, doesn't each thread get a partition?
Isn't that how it works?

Philip

On Wed, Jun 12, 2013 at 9:08 PM, Jun Rao <ju...@gmail.com> wrote:
> Yes, when the consumer is consuming multiple topics, if one thread stops
> consuming topic 1, it can prevent new data getting into the consumer for
> topic 2.
>
> Thanks,
>
> Jun
>
>
> On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <ph...@loggly.com> wrote:
>
>> Hello -- we're using 0.72. We're looking at the source, but want to be
>> sure. :-)
>>
>> We create a single ConsumerConnector, call createMessageStreams, and
>> hand the streams off to individual threads. If one of those threads
>> calls next() on a stream, gets some messages, and then *blocks* in
>> some subsequent operation (and blocks for minutes), can it potentially
>> cause all other threads (calling next() on other streams) to block
>> too? Does something inside the ConsumerConnector block all other
>> stream processing? This would explain some behaviour we're seeing.
>>
>> Thanks,
>>
>> Philip
>>

Re: One 0.72 ConsumerConnector, multiple threads, 1 blocks. What happens?

[Jun Rao <ju...@gmail.com>]

Yes, when the consumer is consuming multiple topics, if one thread stops
consuming topic 1, it can prevent new data getting into the consumer for
topic 2.

Thanks,

Jun


On Wed, Jun 12, 2013 at 7:43 PM, Philip O'Toole <ph...@loggly.com> wrote:

> Hello -- we're using 0.72. We're looking at the source, but want to be
> sure. :-)
>
> We create a single ConsumerConnector, call createMessageStreams, and
> hand the streams off to individual threads. If one of those threads
> calls next() on a stream, gets some messages, and then *blocks* in
> some subsequent operation (and blocks for minutes), can it potentially
> cause all other threads (calling next() on other streams) to block
> too? Does something inside the ConsumerConnector block all other
> stream processing? This would explain some behaviour we're seeing.
>
> Thanks,
>
> Philip
>