You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Paul Mackles <pm...@adobe.com> on 2012/11/15 21:53:15 UTC
QueueFullException
I am tinkering with the java async producer from kafka 7.2 and I am trying to write a client that will never block when sending messages to a kafka broker (dropping them would be better than blocking). I can send messages at a rate that is fast enough to trigger a QueueFullException but the exception never propagates up to my client code. In other words, I see the stack trace and the error message on the console but the call to Producer.send() that triggered the exception just hangs indefinitely. In this setup, queue.enqueueTimeout.ms=0. I get the same result when I set queue.enqueueTimeout.ms to a very low number positive number. Is this the intended behavior? Is there another way to guarantee your client can never block? If I had to guess, maybe the exception is happening in a background thread of some kind and that is why I never see it? My knowledge of scala is extremely limited. Here is the code I am using:
https://gist.github.com/4081183
Thanks,
Paul
Re: QueueFullException
Posted by Paul Mackles <pm...@adobe.com>.
Thanks Joel. That seemed to do it. Since I wasn't handling the exception,
the close() method never got called and some background threads must have
been hanging around, keeping the JVM running. Here is the working producer
code now:
https://gist.github.com/4082757
Since I couldn't find any full examples, I based the code largely on the
ConsoleProducer which also appears to have this problem. I am happy to
submit a patch for the ConsoleProducer and/or also put a full example on
the wiki if that would be helpful.
Thanks,
Paul
On 11/15/12 6:32 PM, "Joel Koshy" <jj...@gmail.com> wrote:
>From the threaddump it looks like your app did get the exception and the
>main thread exited - i.e., it does not appear in the stacktrace. You
>should
>enclose the producer send with a try/catch and finally that closes the
>producer.
>
>
>On Thu, Nov 15, 2012 at 3:19 PM, Paul Mackles <pm...@adobe.com> wrote:
>
>> Thanks. Thread dump from hung producer app here:
>>
>> https://gist.github.com/4082265
>>
>> In case anyone has any insights.
>>
>> On 11/15/12 5:30 PM, "Neha Narkhede" <ne...@gmail.com> wrote:
>>
>> >Paul,
>> >
>> >This behavior is unexpected and the code suggests QueueFullException
>> >is thrown back to the caller. I would take a thread dump when the
>> >producer is blocked to see what's causing the blocking behavior.
>> >
>> >Thanks,
>> >Neha
>> >
>> >On Thu, Nov 15, 2012 at 12:53 PM, Paul Mackles <pm...@adobe.com>
>> wrote:
>> >> I am tinkering with the java async producer from kafka 7.2 and I am
>> >>trying to write a client that will never block when sending messages
>>to
>> >>a kafka broker (dropping them would be better than blocking). I can
>>send
>> >>messages at a rate that is fast enough to trigger a QueueFullException
>> >>but the exception never propagates up to my client code. In other
>>words,
>> >>I see the stack trace and the error message on the console but the
>>call
>> >>to Producer.send() that triggered the exception just hangs
>>indefinitely.
>> >>In this setup, queue.enqueueTimeout.ms=0. I get the same result when I
>> >>set queue.enqueueTimeout.ms to a very low number positive number. Is
>> >>this the intended behavior? Is there another way to guarantee your
>> >>client can never block? If I had to guess, maybe the exception is
>> >>happening in a background thread of some kind and that is why I never
>> >>see it? My knowledge of scala is extremely limited. Here is the code I
>> >>am using:
>> >>
>> >> https://gist.github.com/4081183
>> >>
>> >> Thanks,
>> >> Paul
>> >>
>>
>>
Re: QueueFullException
Posted by Joel Koshy <jj...@gmail.com>.
>From the threaddump it looks like your app did get the exception and the
main thread exited - i.e., it does not appear in the stacktrace. You should
enclose the producer send with a try/catch and finally that closes the
producer.
On Thu, Nov 15, 2012 at 3:19 PM, Paul Mackles <pm...@adobe.com> wrote:
> Thanks. Thread dump from hung producer app here:
>
> https://gist.github.com/4082265
>
> In case anyone has any insights.
>
> On 11/15/12 5:30 PM, "Neha Narkhede" <ne...@gmail.com> wrote:
>
> >Paul,
> >
> >This behavior is unexpected and the code suggests QueueFullException
> >is thrown back to the caller. I would take a thread dump when the
> >producer is blocked to see what's causing the blocking behavior.
> >
> >Thanks,
> >Neha
> >
> >On Thu, Nov 15, 2012 at 12:53 PM, Paul Mackles <pm...@adobe.com>
> wrote:
> >> I am tinkering with the java async producer from kafka 7.2 and I am
> >>trying to write a client that will never block when sending messages to
> >>a kafka broker (dropping them would be better than blocking). I can send
> >>messages at a rate that is fast enough to trigger a QueueFullException
> >>but the exception never propagates up to my client code. In other words,
> >>I see the stack trace and the error message on the console but the call
> >>to Producer.send() that triggered the exception just hangs indefinitely.
> >>In this setup, queue.enqueueTimeout.ms=0. I get the same result when I
> >>set queue.enqueueTimeout.ms to a very low number positive number. Is
> >>this the intended behavior? Is there another way to guarantee your
> >>client can never block? If I had to guess, maybe the exception is
> >>happening in a background thread of some kind and that is why I never
> >>see it? My knowledge of scala is extremely limited. Here is the code I
> >>am using:
> >>
> >> https://gist.github.com/4081183
> >>
> >> Thanks,
> >> Paul
> >>
>
>
Re: QueueFullException
Posted by Paul Mackles <pm...@adobe.com>.
Thanks. Thread dump from hung producer app here:
https://gist.github.com/4082265
In case anyone has any insights.
On 11/15/12 5:30 PM, "Neha Narkhede" <ne...@gmail.com> wrote:
>Paul,
>
>This behavior is unexpected and the code suggests QueueFullException
>is thrown back to the caller. I would take a thread dump when the
>producer is blocked to see what's causing the blocking behavior.
>
>Thanks,
>Neha
>
>On Thu, Nov 15, 2012 at 12:53 PM, Paul Mackles <pm...@adobe.com> wrote:
>> I am tinkering with the java async producer from kafka 7.2 and I am
>>trying to write a client that will never block when sending messages to
>>a kafka broker (dropping them would be better than blocking). I can send
>>messages at a rate that is fast enough to trigger a QueueFullException
>>but the exception never propagates up to my client code. In other words,
>>I see the stack trace and the error message on the console but the call
>>to Producer.send() that triggered the exception just hangs indefinitely.
>>In this setup, queue.enqueueTimeout.ms=0. I get the same result when I
>>set queue.enqueueTimeout.ms to a very low number positive number. Is
>>this the intended behavior? Is there another way to guarantee your
>>client can never block? If I had to guess, maybe the exception is
>>happening in a background thread of some kind and that is why I never
>>see it? My knowledge of scala is extremely limited. Here is the code I
>>am using:
>>
>> https://gist.github.com/4081183
>>
>> Thanks,
>> Paul
>>
Re: QueueFullException
Posted by Neha Narkhede <ne...@gmail.com>.
Paul,
This behavior is unexpected and the code suggests QueueFullException
is thrown back to the caller. I would take a thread dump when the
producer is blocked to see what's causing the blocking behavior.
Thanks,
Neha
On Thu, Nov 15, 2012 at 12:53 PM, Paul Mackles <pm...@adobe.com> wrote:
> I am tinkering with the java async producer from kafka 7.2 and I am trying to write a client that will never block when sending messages to a kafka broker (dropping them would be better than blocking). I can send messages at a rate that is fast enough to trigger a QueueFullException but the exception never propagates up to my client code. In other words, I see the stack trace and the error message on the console but the call to Producer.send() that triggered the exception just hangs indefinitely. In this setup, queue.enqueueTimeout.ms=0. I get the same result when I set queue.enqueueTimeout.ms to a very low number positive number. Is this the intended behavior? Is there another way to guarantee your client can never block? If I had to guess, maybe the exception is happening in a background thread of some kind and that is why I never see it? My knowledge of scala is extremely limited. Here is the code I am using:
>
> https://gist.github.com/4081183
>
> Thanks,
> Paul
>