You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Richard Yu <yo...@gmail.com> on 2019/10/17 23:42:28 UTC
[DISCUSS] KIP-539: Implement mechanism to flush out records in low
volume suppression buffers
Hi all,
I wish to discuss this KIP which would help us in resolving some issues we
have with suppression buffers.
Below is the link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
@John Roesler if you have time, would be great if we could get your input.
Cheers,
Richard
Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in
low volume suppression buffers
Posted by Richard Yu <yo...@gmail.com>.
Hi John,
Reading the KIP over, I actually think that this will be an
excellent extension to what we have now.
I've read the discussion for that KIP and it looks like Jonathan never
mentioned how time would be tracked (wall-clock based that is) while this
KIP I believe proposes a feasible approach to do so.
So in sum, (I will shortly update the KIP), we will probably include
Jonathan's proposal as well.
Hope I could get some further input from you on this. Let me know what you
think. :)
Cheers,
Richard
On Tue, Oct 22, 2019 at 8:01 AM John Roesler <jo...@confluent.io> wrote:
> Hi Richard,
>
> Thanks for the KIP.
>
> After reading it over, I'm wondering if you saw this one:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time
>
> It seems like it overlaps with your use case, where you're willing to
> get events flushed out of the suppression buffer after some wall clock
> time passes, rather than solely based on stream time.
>
> WDYT?
> Thanks,
> -John
>
> On Sun, Oct 20, 2019 at 11:50 AM Richard Yu <yo...@gmail.com>
> wrote:
> >
> > Hi Bill, John,
> >
> > I've made some changes to the KIP after some thinking. I've come up with
> a
> > reasonable solution, I believe, to relieve the problem that is associated
> > with low traffic suppression buffers. Would be great if I can get your
> > input on this. :)
> >
> > Cheers,
> > Richard
> >
> > On Fri, Oct 18, 2019 at 7:16 PM Richard Yu <yo...@gmail.com>
> > wrote:
> >
> > > Hi Bill,
> > >
> > > Thanks for the input!
> > > TBH, I am think that suppression buffers are not used *in response *to
> > > low traffic conditions.
> > > Rather, we are trying to fix the situation when low traffic conditions
> > > occur in a suppression buffer (for example, previously, the same
> > > suppression buffer had a decent volume of records entering it, thus
> > > advancing the stream time).
> > > In summary, when these conditions do hit, we want to advance the stream
> > > time somehow to resolve this issue.
> > > Reflecting on this though, I am not completely certain if this really
> > > stops us from implementing per key stream time tracking because the
> problem
> > > wouldn't be made *that *much worse.
> > >
> > > Cheers,
> > > Richard
> > >
> > > On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bb...@gmail.com>
> wrote:
> > >
> > >> Hi Richard,
> > >>
> > >> Thanks for the KIP proposal. I understand the situation you are
> > >> describing.
> > >> But in my mind, if there is a low traffic condition and you need to
> keep
> > >> records going downstream at regular intervals, I'm wondering if using
> > >> suppression is the correct approach.
> > >> IMHO it seems it would be better to use the PAPI or a Transform on
> the DSL
> > >> with a scheduled punctuation call.
> > >>
> > >> Just my 2 cents.
> > >>
> > >> Thanks,
> > >> Bill
> > >>
> > >> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <
> yohan.richard.yu@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > I wish to discuss this KIP which would help us in resolving some
> issues
> > >> we
> > >> > have with suppression buffers.
> > >> > Below is the link:
> > >> >
> > >> >
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
> > >> >
> > >> > @John Roesler if you have time, would be great if we could get your
> > >> input.
> > >> >
> > >> > Cheers,
> > >> > Richard
> > >> >
> > >>
> > >
>
Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in
low volume suppression buffers
Posted by John Roesler <jo...@confluent.io>.
Hi Richard,
Thanks for the KIP.
After reading it over, I'm wondering if you saw this one:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time
It seems like it overlaps with your use case, where you're willing to
get events flushed out of the suppression buffer after some wall clock
time passes, rather than solely based on stream time.
WDYT?
Thanks,
-John
On Sun, Oct 20, 2019 at 11:50 AM Richard Yu <yo...@gmail.com> wrote:
>
> Hi Bill, John,
>
> I've made some changes to the KIP after some thinking. I've come up with a
> reasonable solution, I believe, to relieve the problem that is associated
> with low traffic suppression buffers. Would be great if I can get your
> input on this. :)
>
> Cheers,
> Richard
>
> On Fri, Oct 18, 2019 at 7:16 PM Richard Yu <yo...@gmail.com>
> wrote:
>
> > Hi Bill,
> >
> > Thanks for the input!
> > TBH, I am think that suppression buffers are not used *in response *to
> > low traffic conditions.
> > Rather, we are trying to fix the situation when low traffic conditions
> > occur in a suppression buffer (for example, previously, the same
> > suppression buffer had a decent volume of records entering it, thus
> > advancing the stream time).
> > In summary, when these conditions do hit, we want to advance the stream
> > time somehow to resolve this issue.
> > Reflecting on this though, I am not completely certain if this really
> > stops us from implementing per key stream time tracking because the problem
> > wouldn't be made *that *much worse.
> >
> > Cheers,
> > Richard
> >
> > On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bb...@gmail.com> wrote:
> >
> >> Hi Richard,
> >>
> >> Thanks for the KIP proposal. I understand the situation you are
> >> describing.
> >> But in my mind, if there is a low traffic condition and you need to keep
> >> records going downstream at regular intervals, I'm wondering if using
> >> suppression is the correct approach.
> >> IMHO it seems it would be better to use the PAPI or a Transform on the DSL
> >> with a scheduled punctuation call.
> >>
> >> Just my 2 cents.
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yo...@gmail.com>
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I wish to discuss this KIP which would help us in resolving some issues
> >> we
> >> > have with suppression buffers.
> >> > Below is the link:
> >> >
> >> >
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
> >> >
> >> > @John Roesler if you have time, would be great if we could get your
> >> input.
> >> >
> >> > Cheers,
> >> > Richard
> >> >
> >>
> >
Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in
low volume suppression buffers
Posted by Richard Yu <yo...@gmail.com>.
Hi Bill, John,
I've made some changes to the KIP after some thinking. I've come up with a
reasonable solution, I believe, to relieve the problem that is associated
with low traffic suppression buffers. Would be great if I can get your
input on this. :)
Cheers,
Richard
On Fri, Oct 18, 2019 at 7:16 PM Richard Yu <yo...@gmail.com>
wrote:
> Hi Bill,
>
> Thanks for the input!
> TBH, I am think that suppression buffers are not used *in response *to
> low traffic conditions.
> Rather, we are trying to fix the situation when low traffic conditions
> occur in a suppression buffer (for example, previously, the same
> suppression buffer had a decent volume of records entering it, thus
> advancing the stream time).
> In summary, when these conditions do hit, we want to advance the stream
> time somehow to resolve this issue.
> Reflecting on this though, I am not completely certain if this really
> stops us from implementing per key stream time tracking because the problem
> wouldn't be made *that *much worse.
>
> Cheers,
> Richard
>
> On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bb...@gmail.com> wrote:
>
>> Hi Richard,
>>
>> Thanks for the KIP proposal. I understand the situation you are
>> describing.
>> But in my mind, if there is a low traffic condition and you need to keep
>> records going downstream at regular intervals, I'm wondering if using
>> suppression is the correct approach.
>> IMHO it seems it would be better to use the PAPI or a Transform on the DSL
>> with a scheduled punctuation call.
>>
>> Just my 2 cents.
>>
>> Thanks,
>> Bill
>>
>> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yo...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > I wish to discuss this KIP which would help us in resolving some issues
>> we
>> > have with suppression buffers.
>> > Below is the link:
>> >
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
>> >
>> > @John Roesler if you have time, would be great if we could get your
>> input.
>> >
>> > Cheers,
>> > Richard
>> >
>>
>
Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in
low volume suppression buffers
Posted by Richard Yu <yo...@gmail.com>.
Hi Bill,
Thanks for the input!
TBH, I am think that suppression buffers are not used *in response *to low
traffic conditions.
Rather, we are trying to fix the situation when low traffic conditions
occur in a suppression buffer (for example, previously, the same
suppression buffer had a decent volume of records entering it, thus
advancing the stream time).
In summary, when these conditions do hit, we want to advance the stream
time somehow to resolve this issue.
Reflecting on this though, I am not completely certain if this really stops
us from implementing per key stream time tracking because the problem
wouldn't be made *that *much worse.
Cheers,
Richard
On Fri, Oct 18, 2019 at 11:46 AM Bill Bejeck <bb...@gmail.com> wrote:
> Hi Richard,
>
> Thanks for the KIP proposal. I understand the situation you are
> describing.
> But in my mind, if there is a low traffic condition and you need to keep
> records going downstream at regular intervals, I'm wondering if using
> suppression is the correct approach.
> IMHO it seems it would be better to use the PAPI or a Transform on the DSL
> with a scheduled punctuation call.
>
> Just my 2 cents.
>
> Thanks,
> Bill
>
> On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yo...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I wish to discuss this KIP which would help us in resolving some issues
> we
> > have with suppression buffers.
> > Below is the link:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
> >
> > @John Roesler if you have time, would be great if we could get your
> input.
> >
> > Cheers,
> > Richard
> >
>
Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in
low volume suppression buffers
Posted by Bill Bejeck <bb...@gmail.com>.
Hi Richard,
Thanks for the KIP proposal. I understand the situation you are
describing.
But in my mind, if there is a low traffic condition and you need to keep
records going downstream at regular intervals, I'm wondering if using
suppression is the correct approach.
IMHO it seems it would be better to use the PAPI or a Transform on the DSL
with a scheduled punctuation call.
Just my 2 cents.
Thanks,
Bill
On Thu, Oct 17, 2019 at 7:42 PM Richard Yu <yo...@gmail.com>
wrote:
> Hi all,
>
> I wish to discuss this KIP which would help us in resolving some issues we
> have with suppression buffers.
> Below is the link:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers
>
> @John Roesler if you have time, would be great if we could get your input.
>
> Cheers,
> Richard
>