You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Alexey Romanchuk <al...@gmail.com> on 2016/03/14 16:18:14 UTC
New client commitAsync SendFailedException
Hi all!
I am using new client 0.9.0.1.
I found that when I call commitAsync multiple times before calling poll
most of commits failed with SendFailedException.
Here it is an example of code -
https://gist.github.com/13h3r/42633bcd64b80ddffe6b
Could you please explain commitAsync in more details?
What is wrong with calling commitAsync multiple times? Should I call poll
immediate after calling commitAsync? Should I precalculate offsets for
multiple commits for every topic/partition to minimize number of offsets?
Thanks!
Re: New client commitAsync SendFailedException
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Alexey, looks like we didn't have a JIRA before, but I created one just
now: https://issues.apache.org/jira/browse/KAFKA-3412. As soon as I get a
little time, I'll polish up my old patch and submit it.
-Jason
On Wed, Mar 16, 2016 at 4:10 AM, Alexey Romanchuk <
alexey.romanchuk@gmail.com> wrote:
> Jason, Jay, could you please share bug number to track this issue?
>
> On Tue, Mar 15, 2016 at 10:57 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Yeah, I agree the bug is probably more serious than I had thought before
> > (I've gotten too used to examples with only a single commitAsync()
> call). I
> > had worked on a patch to fix this at one point. Let me see if I can dig
> it
> > up.
> >
> > -Jason
> >
> > On Tue, Mar 15, 2016 at 1:06 AM, Alexey Romanchuk <
> > alexey.romanchuk@gmail.com> wrote:
> >
> > > Jay, I am completely agreed with you. From API level it is nothing I
> can
> > do
> > > about it. Anything else I have to do with this issue? Reproducible
> > example
> > > here - https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> > >
> > > Thanks!
> > >
> > > On Tue, Mar 15, 2016 at 11:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> > >
> > > > This seems like a bug, no? It should just initiate the request not
> wait
> > > for
> > > > it to be written, there is no way for the user to reason about the
> > state
> > > of
> > > > the send buffer.
> > > >
> > > > -jay
> > > >
> > > > On Monday, March 14, 2016, Jason Gustafson <ja...@confluent.io>
> wrote:
> > > >
> > > > > Hey Alexey,
> > > > >
> > > > > Asynchronous commit handling could probably be improved quite a
> bit.
> > > > > Basically what's happening is that the client's send buffer is
> > getting
> > > > > filled up, which causes the subsequent commits to fail with
> > > > > SendFailedException. We don't currently implement any retrying for
> > > > > asynchronous commits, so even transient failures get bubbled up to
> > the
> > > > > user. This is.. unfortunate. We probably can implement some retry
> > > logic,
> > > > > but we need to be careful careful about preserving the commit
> order.
> > > > Maybe
> > > > > this is as simple as adding sequence numbers for each partition. I
> > > > haven't
> > > > > thought it all the way through, but you're welcome to open a JIRA
> if
> > > you
> > > > > think it would be helpful.
> > > > >
> > > > > For now, the easiest way to mitigate the problem is probably to
> batch
> > > the
> > > > > commits into a single call to commitAsync(). Would that work for
> you?
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> > > > > alexey.romanchuk@gmail.com <javascript:;>> wrote:
> > > > >
> > > > > > Hi all!
> > > > > >
> > > > > > I am using new client 0.9.0.1.
> > > > > >
> > > > > > I found that when I call commitAsync multiple times before
> calling
> > > poll
> > > > > > most of commits failed with SendFailedException.
> > > > > >
> > > > > > Here it is an example of code -
> > > > > > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> > > > > >
> > > > > > Could you please explain commitAsync in more details?
> > > > > >
> > > > > > What is wrong with calling commitAsync multiple times? Should I
> > call
> > > > poll
> > > > > > immediate after calling commitAsync? Should I precalculate
> offsets
> > > for
> > > > > > multiple commits for every topic/partition to minimize number of
> > > > offsets?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New client commitAsync SendFailedException
Posted by Alexey Romanchuk <al...@gmail.com>.
Jason, Jay, could you please share bug number to track this issue?
On Tue, Mar 15, 2016 at 10:57 PM, Jason Gustafson <ja...@confluent.io>
wrote:
> Yeah, I agree the bug is probably more serious than I had thought before
> (I've gotten too used to examples with only a single commitAsync() call). I
> had worked on a patch to fix this at one point. Let me see if I can dig it
> up.
>
> -Jason
>
> On Tue, Mar 15, 2016 at 1:06 AM, Alexey Romanchuk <
> alexey.romanchuk@gmail.com> wrote:
>
> > Jay, I am completely agreed with you. From API level it is nothing I can
> do
> > about it. Anything else I have to do with this issue? Reproducible
> example
> > here - https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> >
> > Thanks!
> >
> > On Tue, Mar 15, 2016 at 11:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > This seems like a bug, no? It should just initiate the request not wait
> > for
> > > it to be written, there is no way for the user to reason about the
> state
> > of
> > > the send buffer.
> > >
> > > -jay
> > >
> > > On Monday, March 14, 2016, Jason Gustafson <ja...@confluent.io> wrote:
> > >
> > > > Hey Alexey,
> > > >
> > > > Asynchronous commit handling could probably be improved quite a bit.
> > > > Basically what's happening is that the client's send buffer is
> getting
> > > > filled up, which causes the subsequent commits to fail with
> > > > SendFailedException. We don't currently implement any retrying for
> > > > asynchronous commits, so even transient failures get bubbled up to
> the
> > > > user. This is.. unfortunate. We probably can implement some retry
> > logic,
> > > > but we need to be careful careful about preserving the commit order.
> > > Maybe
> > > > this is as simple as adding sequence numbers for each partition. I
> > > haven't
> > > > thought it all the way through, but you're welcome to open a JIRA if
> > you
> > > > think it would be helpful.
> > > >
> > > > For now, the easiest way to mitigate the problem is probably to batch
> > the
> > > > commits into a single call to commitAsync(). Would that work for you?
> > > >
> > > > -Jason
> > > >
> > > > On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> > > > alexey.romanchuk@gmail.com <javascript:;>> wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > I am using new client 0.9.0.1.
> > > > >
> > > > > I found that when I call commitAsync multiple times before calling
> > poll
> > > > > most of commits failed with SendFailedException.
> > > > >
> > > > > Here it is an example of code -
> > > > > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> > > > >
> > > > > Could you please explain commitAsync in more details?
> > > > >
> > > > > What is wrong with calling commitAsync multiple times? Should I
> call
> > > poll
> > > > > immediate after calling commitAsync? Should I precalculate offsets
> > for
> > > > > multiple commits for every topic/partition to minimize number of
> > > offsets?
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > >
> >
>
Re: New client commitAsync SendFailedException
Posted by Jason Gustafson <ja...@confluent.io>.
Yeah, I agree the bug is probably more serious than I had thought before
(I've gotten too used to examples with only a single commitAsync() call). I
had worked on a patch to fix this at one point. Let me see if I can dig it
up.
-Jason
On Tue, Mar 15, 2016 at 1:06 AM, Alexey Romanchuk <
alexey.romanchuk@gmail.com> wrote:
> Jay, I am completely agreed with you. From API level it is nothing I can do
> about it. Anything else I have to do with this issue? Reproducible example
> here - https://gist.github.com/13h3r/42633bcd64b80ddffe6b
>
> Thanks!
>
> On Tue, Mar 15, 2016 at 11:24 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > This seems like a bug, no? It should just initiate the request not wait
> for
> > it to be written, there is no way for the user to reason about the state
> of
> > the send buffer.
> >
> > -jay
> >
> > On Monday, March 14, 2016, Jason Gustafson <ja...@confluent.io> wrote:
> >
> > > Hey Alexey,
> > >
> > > Asynchronous commit handling could probably be improved quite a bit.
> > > Basically what's happening is that the client's send buffer is getting
> > > filled up, which causes the subsequent commits to fail with
> > > SendFailedException. We don't currently implement any retrying for
> > > asynchronous commits, so even transient failures get bubbled up to the
> > > user. This is.. unfortunate. We probably can implement some retry
> logic,
> > > but we need to be careful careful about preserving the commit order.
> > Maybe
> > > this is as simple as adding sequence numbers for each partition. I
> > haven't
> > > thought it all the way through, but you're welcome to open a JIRA if
> you
> > > think it would be helpful.
> > >
> > > For now, the easiest way to mitigate the problem is probably to batch
> the
> > > commits into a single call to commitAsync(). Would that work for you?
> > >
> > > -Jason
> > >
> > > On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> > > alexey.romanchuk@gmail.com <javascript:;>> wrote:
> > >
> > > > Hi all!
> > > >
> > > > I am using new client 0.9.0.1.
> > > >
> > > > I found that when I call commitAsync multiple times before calling
> poll
> > > > most of commits failed with SendFailedException.
> > > >
> > > > Here it is an example of code -
> > > > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> > > >
> > > > Could you please explain commitAsync in more details?
> > > >
> > > > What is wrong with calling commitAsync multiple times? Should I call
> > poll
> > > > immediate after calling commitAsync? Should I precalculate offsets
> for
> > > > multiple commits for every topic/partition to minimize number of
> > offsets?
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
Re: New client commitAsync SendFailedException
Posted by Alexey Romanchuk <al...@gmail.com>.
Jay, I am completely agreed with you. From API level it is nothing I can do
about it. Anything else I have to do with this issue? Reproducible example
here - https://gist.github.com/13h3r/42633bcd64b80ddffe6b
Thanks!
On Tue, Mar 15, 2016 at 11:24 AM, Jay Kreps <ja...@confluent.io> wrote:
> This seems like a bug, no? It should just initiate the request not wait for
> it to be written, there is no way for the user to reason about the state of
> the send buffer.
>
> -jay
>
> On Monday, March 14, 2016, Jason Gustafson <ja...@confluent.io> wrote:
>
> > Hey Alexey,
> >
> > Asynchronous commit handling could probably be improved quite a bit.
> > Basically what's happening is that the client's send buffer is getting
> > filled up, which causes the subsequent commits to fail with
> > SendFailedException. We don't currently implement any retrying for
> > asynchronous commits, so even transient failures get bubbled up to the
> > user. This is.. unfortunate. We probably can implement some retry logic,
> > but we need to be careful careful about preserving the commit order.
> Maybe
> > this is as simple as adding sequence numbers for each partition. I
> haven't
> > thought it all the way through, but you're welcome to open a JIRA if you
> > think it would be helpful.
> >
> > For now, the easiest way to mitigate the problem is probably to batch the
> > commits into a single call to commitAsync(). Would that work for you?
> >
> > -Jason
> >
> > On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> > alexey.romanchuk@gmail.com <javascript:;>> wrote:
> >
> > > Hi all!
> > >
> > > I am using new client 0.9.0.1.
> > >
> > > I found that when I call commitAsync multiple times before calling poll
> > > most of commits failed with SendFailedException.
> > >
> > > Here it is an example of code -
> > > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> > >
> > > Could you please explain commitAsync in more details?
> > >
> > > What is wrong with calling commitAsync multiple times? Should I call
> poll
> > > immediate after calling commitAsync? Should I precalculate offsets for
> > > multiple commits for every topic/partition to minimize number of
> offsets?
> > >
> > > Thanks!
> > >
> >
>
Re: New client commitAsync SendFailedException
Posted by Jay Kreps <ja...@confluent.io>.
This seems like a bug, no? It should just initiate the request not wait for
it to be written, there is no way for the user to reason about the state of
the send buffer.
-jay
On Monday, March 14, 2016, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Alexey,
>
> Asynchronous commit handling could probably be improved quite a bit.
> Basically what's happening is that the client's send buffer is getting
> filled up, which causes the subsequent commits to fail with
> SendFailedException. We don't currently implement any retrying for
> asynchronous commits, so even transient failures get bubbled up to the
> user. This is.. unfortunate. We probably can implement some retry logic,
> but we need to be careful careful about preserving the commit order. Maybe
> this is as simple as adding sequence numbers for each partition. I haven't
> thought it all the way through, but you're welcome to open a JIRA if you
> think it would be helpful.
>
> For now, the easiest way to mitigate the problem is probably to batch the
> commits into a single call to commitAsync(). Would that work for you?
>
> -Jason
>
> On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> alexey.romanchuk@gmail.com <javascript:;>> wrote:
>
> > Hi all!
> >
> > I am using new client 0.9.0.1.
> >
> > I found that when I call commitAsync multiple times before calling poll
> > most of commits failed with SendFailedException.
> >
> > Here it is an example of code -
> > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> >
> > Could you please explain commitAsync in more details?
> >
> > What is wrong with calling commitAsync multiple times? Should I call poll
> > immediate after calling commitAsync? Should I precalculate offsets for
> > multiple commits for every topic/partition to minimize number of offsets?
> >
> > Thanks!
> >
>
Re: New client commitAsync SendFailedException
Posted by Alexey Romanchuk <al...@gmail.com>.
Thanks for reply Jason!
Is it any way to control size of this buffer? Will it fails if I try to
commit offsets for 100 topics/partitions?
Sure I can work around it by batching all commits into one call, but the
problem I can see is that the API does not enforce me to do this. From
client point of view I did some correct stuff and got the exception (even
more - asynchronous exception) with no idea what was wrong in my actions.
On Tue, Mar 15, 2016 at 1:00 AM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Alexey,
>
> Asynchronous commit handling could probably be improved quite a bit.
> Basically what's happening is that the client's send buffer is getting
> filled up, which causes the subsequent commits to fail with
> SendFailedException. We don't currently implement any retrying for
> asynchronous commits, so even transient failures get bubbled up to the
> user. This is.. unfortunate. We probably can implement some retry logic,
> but we need to be careful careful about preserving the commit order. Maybe
> this is as simple as adding sequence numbers for each partition. I haven't
> thought it all the way through, but you're welcome to open a JIRA if you
> think it would be helpful.
>
> For now, the easiest way to mitigate the problem is probably to batch the
> commits into a single call to commitAsync(). Would that work for you?
>
> -Jason
>
> On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
> alexey.romanchuk@gmail.com> wrote:
>
> > Hi all!
> >
> > I am using new client 0.9.0.1.
> >
> > I found that when I call commitAsync multiple times before calling poll
> > most of commits failed with SendFailedException.
> >
> > Here it is an example of code -
> > https://gist.github.com/13h3r/42633bcd64b80ddffe6b
> >
> > Could you please explain commitAsync in more details?
> >
> > What is wrong with calling commitAsync multiple times? Should I call poll
> > immediate after calling commitAsync? Should I precalculate offsets for
> > multiple commits for every topic/partition to minimize number of offsets?
> >
> > Thanks!
> >
>
Re: New client commitAsync SendFailedException
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Alexey,
Asynchronous commit handling could probably be improved quite a bit.
Basically what's happening is that the client's send buffer is getting
filled up, which causes the subsequent commits to fail with
SendFailedException. We don't currently implement any retrying for
asynchronous commits, so even transient failures get bubbled up to the
user. This is.. unfortunate. We probably can implement some retry logic,
but we need to be careful careful about preserving the commit order. Maybe
this is as simple as adding sequence numbers for each partition. I haven't
thought it all the way through, but you're welcome to open a JIRA if you
think it would be helpful.
For now, the easiest way to mitigate the problem is probably to batch the
commits into a single call to commitAsync(). Would that work for you?
-Jason
On Mon, Mar 14, 2016 at 8:18 AM, Alexey Romanchuk <
alexey.romanchuk@gmail.com> wrote:
> Hi all!
>
> I am using new client 0.9.0.1.
>
> I found that when I call commitAsync multiple times before calling poll
> most of commits failed with SendFailedException.
>
> Here it is an example of code -
> https://gist.github.com/13h3r/42633bcd64b80ddffe6b
>
> Could you please explain commitAsync in more details?
>
> What is wrong with calling commitAsync multiple times? Should I call poll
> immediate after calling commitAsync? Should I precalculate offsets for
> multiple commits for every topic/partition to minimize number of offsets?
>
> Thanks!
>