You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Steven Wu <st...@gmail.com> on 2015/02/10 02:19:08 UTC
could new java producer miss callbacks after successful send?
We observed some small discrepancy in messages sent per second reported at
different points. 1) and 4) matches very close. 2) and 3) matches very
close but are about *5-6% lower* compared to 1) and 4).
1) send attempt from producer
2) send success from producer
3) record-send-rate reported by kafka producer
4) MessagesInPerSecond reported by kafka broker
note that send success for 2) is incremented when onCompletion is called
without error/exception). there is also a send error count when
onCompletion is called with error. it is always zero.
that's why I am wondering whether there are some callback misses?
some info about the setup
* producer: 0.8.2-beta
* broker: 0.8.1.1
* acks=1
Thanks,
Steven
Re: could new java producer miss callbacks after successful send?
Posted by Jay Kreps <ja...@gmail.com>.
Cool, let us know if you can uncover anything as that would be a serious
issue if we are dropping callbacks.
-Jay
On Tue, Feb 10, 2015 at 6:44 PM, Steven Wu <st...@gmail.com> wrote:
> couldn't reproduce/confirm the issue with my test. send 6 million msgs from
> 6 instances. got 6 million callbacks.
>
> this could be some metric issues.
>
> On Mon, Feb 9, 2015 at 8:23 PM, Steven Wu <st...@gmail.com> wrote:
>
> > I don't have strong evidence that this is a bug yet. let me write some
> > test program and see if I can confirm/reproduce the issue.
> >
> > On Mon, Feb 9, 2015 at 7:59 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> >> Hmm, that does sound like a bug, we haven't seen that. How easy is it to
> >> reproduce this?
> >>
> >> -Jay
> >>
> >> On Mon, Feb 9, 2015 at 5:19 PM, Steven Wu <st...@gmail.com> wrote:
> >>
> >> > We observed some small discrepancy in messages sent per second
> reported
> >> at
> >> > different points. 1) and 4) matches very close. 2) and 3) matches very
> >> > close but are about *5-6% lower* compared to 1) and 4).
> >> > 1) send attempt from producer
> >> > 2) send success from producer
> >> > 3) record-send-rate reported by kafka producer
> >> > 4) MessagesInPerSecond reported by kafka broker
> >> >
> >> > note that send success for 2) is incremented when onCompletion is
> called
> >> > without error/exception). there is also a send error count when
> >> > onCompletion is called with error. it is always zero.
> >> >
> >> > that's why I am wondering whether there are some callback misses?
> >> >
> >> > some info about the setup
> >> > * producer: 0.8.2-beta
> >> > * broker: 0.8.1.1
> >> > * acks=1
> >> >
> >> > Thanks,
> >> > Steven
> >> >
> >>
> >
> >
>
Re: could new java producer miss callbacks after successful send?
Posted by Steven Wu <st...@gmail.com>.
couldn't reproduce/confirm the issue with my test. send 6 million msgs from
6 instances. got 6 million callbacks.
this could be some metric issues.
On Mon, Feb 9, 2015 at 8:23 PM, Steven Wu <st...@gmail.com> wrote:
> I don't have strong evidence that this is a bug yet. let me write some
> test program and see if I can confirm/reproduce the issue.
>
> On Mon, Feb 9, 2015 at 7:59 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Hmm, that does sound like a bug, we haven't seen that. How easy is it to
>> reproduce this?
>>
>> -Jay
>>
>> On Mon, Feb 9, 2015 at 5:19 PM, Steven Wu <st...@gmail.com> wrote:
>>
>> > We observed some small discrepancy in messages sent per second reported
>> at
>> > different points. 1) and 4) matches very close. 2) and 3) matches very
>> > close but are about *5-6% lower* compared to 1) and 4).
>> > 1) send attempt from producer
>> > 2) send success from producer
>> > 3) record-send-rate reported by kafka producer
>> > 4) MessagesInPerSecond reported by kafka broker
>> >
>> > note that send success for 2) is incremented when onCompletion is called
>> > without error/exception). there is also a send error count when
>> > onCompletion is called with error. it is always zero.
>> >
>> > that's why I am wondering whether there are some callback misses?
>> >
>> > some info about the setup
>> > * producer: 0.8.2-beta
>> > * broker: 0.8.1.1
>> > * acks=1
>> >
>> > Thanks,
>> > Steven
>> >
>>
>
>
Re: could new java producer miss callbacks after successful send?
Posted by Steven Wu <st...@gmail.com>.
I don't have strong evidence that this is a bug yet. let me write some test
program and see if I can confirm/reproduce the issue.
On Mon, Feb 9, 2015 at 7:59 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hmm, that does sound like a bug, we haven't seen that. How easy is it to
> reproduce this?
>
> -Jay
>
> On Mon, Feb 9, 2015 at 5:19 PM, Steven Wu <st...@gmail.com> wrote:
>
> > We observed some small discrepancy in messages sent per second reported
> at
> > different points. 1) and 4) matches very close. 2) and 3) matches very
> > close but are about *5-6% lower* compared to 1) and 4).
> > 1) send attempt from producer
> > 2) send success from producer
> > 3) record-send-rate reported by kafka producer
> > 4) MessagesInPerSecond reported by kafka broker
> >
> > note that send success for 2) is incremented when onCompletion is called
> > without error/exception). there is also a send error count when
> > onCompletion is called with error. it is always zero.
> >
> > that's why I am wondering whether there are some callback misses?
> >
> > some info about the setup
> > * producer: 0.8.2-beta
> > * broker: 0.8.1.1
> > * acks=1
> >
> > Thanks,
> > Steven
> >
>
Re: could new java producer miss callbacks after successful send?
Posted by Jay Kreps <ja...@gmail.com>.
Hmm, that does sound like a bug, we haven't seen that. How easy is it to
reproduce this?
-Jay
On Mon, Feb 9, 2015 at 5:19 PM, Steven Wu <st...@gmail.com> wrote:
> We observed some small discrepancy in messages sent per second reported at
> different points. 1) and 4) matches very close. 2) and 3) matches very
> close but are about *5-6% lower* compared to 1) and 4).
> 1) send attempt from producer
> 2) send success from producer
> 3) record-send-rate reported by kafka producer
> 4) MessagesInPerSecond reported by kafka broker
>
> note that send success for 2) is incremented when onCompletion is called
> without error/exception). there is also a send error count when
> onCompletion is called with error. it is always zero.
>
> that's why I am wondering whether there are some callback misses?
>
> some info about the setup
> * producer: 0.8.2-beta
> * broker: 0.8.1.1
> * acks=1
>
> Thanks,
> Steven
>