You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by William Bottrell <bo...@gmail.com> on 2020/07/07 18:48:15 UTC
[VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Hi everyone,
I'd like to start a vote for adding two new time API's to ProcessorContext.
Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext>
Thanks everyone for the initial feedback and thanks for your time.
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Jorge Esteban Quilcate Otoya <qu...@gmail.com>.
Yet another quick FYI.
While implementing KIP-820, we found that `api.MockProcessorContext` was
missing these new methods as well.
We added the new method to the new `api.MockProcessorContext` via
https://issues.apache.org/jira/browse/KAFKA-13654.
Please let us know if there are any concerns.
I updated the KIP accordingly.
Cheers,
Jorge
On Tue, 15 Mar 2022 at 23:13, Matthias J. Sax <mj...@apache.org> wrote:
> Just a quick FYI.
>
> KIP-622 overlapped with KIP-478.
>
> We added the new method to the new `api.ProcessorContext` via
> https://issues.apache.org/jira/browse/KAFKA-13699 for 3.2.0 release.
>
> Please let us know if there are any concerns.
>
> I updated the KIP accordingly.
>
> -Matthias
>
> On 3/5/21 8:42 PM, Rohit Deshpande wrote:
> > Hello all,
> > Based on the feedback of the pr <
> https://github.com/apache/kafka/pull/9744>
> > https://github.com/apache/kafka/pull/9744, there are following changes
> done
> > to the kip
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> >
> > .
> >
> > *ProcessorContext#currentSystemTimeMs()*
> >
> > It is expected that this method will return the internally cached system
> > timestamp from the Kafka Stream runtime. Thus, it may return a different
> > value compared to System.currentTimeMillis(). The cached system time
> > represents the time when we start processing / punctuating, and it would
> > not change throughout the process / punctuate. So this method will return
> > current system time (also called wall-clock time) known from kafka
> streams
> > runtime.
> >
> > New methods to MockProcessorContext for testing purposes:
> >
> > *MockProcessorContext#setRecordTimestamp*: set record timestamp
> >
> > *MockProcessorContext#setCurrentSystemTimeMs:* set system timestamp
> >
> > *MockProcessorContext#setCurrentStreamTimeMs*: set stream time
> >
> > Deprecate method: MockProcessorContext#setTimestamp as it's name is
> > misleading and we are adding a new method
> > MockProcessorContext#setRecordTimestamp which does the same work.
> >
> > Please let me know if you have any thoughts or concerns with this change.
> >
> > Thanks,
> > Roohit
> >
> > On Fri, Dec 4, 2020 at 7:31 PM Rohit Deshpande <ro...@gmail.com>
> > wrote:
> >
> >> Hello all,
> >> I am closing the vote for this KIP:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> >>
> >> Summary of the KIP:
> >> Planning to add two new methods to ProcessorContext:
> >> 1. long currentSystemTimeMs() to fetch wall-clock time
> >> 2. long currentStreamTimeMs() to fetch maximum timestamp of any record
> yet
> >> processed by the task
> >>
> >> Thanks,
> >> Rohit
> >>
> >>
> >> On 2020/12/01 16:09:54, Bill Bejeck <bb...@gmail.com> wrote:
> >>> Sorry for jumping into this so late,
> >>>
> >>> Thanks for the KIP, I'm a +1 (binding)
> >>>
> >>> -Bill
> >>>
> >>> On Sun, Jul 26, 2020 at 11:06 AM John Roesler <jo...@vvcephei.org>
> wrote:
> >>>
> >>>> Thanks William,
> >>>>
> >>>> I’m +1 (binding)
> >>>>
> >>>> Thanks,
> >>>> John
> >>>>
> >>>> On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> >>>>> Thanks all, +1 (non-binding)
> >>>>>
> >>>>> Cheers,
> >>>>> Sophie
> >>>>>
> >>>>> On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io>
> >> wrote:
> >>>>>
> >>>>>> Thanks Will and Piotr,
> >>>>>>
> >>>>>> +1 (non-binding)
> >>>>>>
> >>>>>> Best,
> >>>>>> Bruno
> >>>>>>
> >>>>>> On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Thanks for the KIP.
> >>>>>>>
> >>>>>>> +1 (binding)
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 7/7/20 11:48 AM, William Bottrell wrote:
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> I'd like to start a vote for adding two new time API's to
> >>>>>> ProcessorContext.
> >>>>>>>>
> >>>>>>>> Add currentSystemTimeMs and currentStreamTimeMs to
> >> ProcessorContext
> >>>>>>>> <
> >>>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Thanks everyone for the initial feedback and thanks for your
> >> time.
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by "Matthias J. Sax" <mj...@apache.org>.
Just a quick FYI.
KIP-622 overlapped with KIP-478.
We added the new method to the new `api.ProcessorContext` via
https://issues.apache.org/jira/browse/KAFKA-13699 for 3.2.0 release.
Please let us know if there are any concerns.
I updated the KIP accordingly.
-Matthias
On 3/5/21 8:42 PM, Rohit Deshpande wrote:
> Hello all,
> Based on the feedback of the pr <https://github.com/apache/kafka/pull/9744>
> https://github.com/apache/kafka/pull/9744, there are following changes done
> to the kip
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext>
> .
>
> *ProcessorContext#currentSystemTimeMs()*
>
> It is expected that this method will return the internally cached system
> timestamp from the Kafka Stream runtime. Thus, it may return a different
> value compared to System.currentTimeMillis(). The cached system time
> represents the time when we start processing / punctuating, and it would
> not change throughout the process / punctuate. So this method will return
> current system time (also called wall-clock time) known from kafka streams
> runtime.
>
> New methods to MockProcessorContext for testing purposes:
>
> *MockProcessorContext#setRecordTimestamp*: set record timestamp
>
> *MockProcessorContext#setCurrentSystemTimeMs:* set system timestamp
>
> *MockProcessorContext#setCurrentStreamTimeMs*: set stream time
>
> Deprecate method: MockProcessorContext#setTimestamp as it's name is
> misleading and we are adding a new method
> MockProcessorContext#setRecordTimestamp which does the same work.
>
> Please let me know if you have any thoughts or concerns with this change.
>
> Thanks,
> Roohit
>
> On Fri, Dec 4, 2020 at 7:31 PM Rohit Deshpande <ro...@gmail.com>
> wrote:
>
>> Hello all,
>> I am closing the vote for this KIP:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
>>
>> Summary of the KIP:
>> Planning to add two new methods to ProcessorContext:
>> 1. long currentSystemTimeMs() to fetch wall-clock time
>> 2. long currentStreamTimeMs() to fetch maximum timestamp of any record yet
>> processed by the task
>>
>> Thanks,
>> Rohit
>>
>>
>> On 2020/12/01 16:09:54, Bill Bejeck <bb...@gmail.com> wrote:
>>> Sorry for jumping into this so late,
>>>
>>> Thanks for the KIP, I'm a +1 (binding)
>>>
>>> -Bill
>>>
>>> On Sun, Jul 26, 2020 at 11:06 AM John Roesler <jo...@vvcephei.org> wrote:
>>>
>>>> Thanks William,
>>>>
>>>> I’m +1 (binding)
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
>>>>> Thanks all, +1 (non-binding)
>>>>>
>>>>> Cheers,
>>>>> Sophie
>>>>>
>>>>> On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io>
>> wrote:
>>>>>
>>>>>> Thanks Will and Piotr,
>>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> Best,
>>>>>> Bruno
>>>>>>
>>>>>> On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>> Thanks for the KIP.
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 7/7/20 11:48 AM, William Bottrell wrote:
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I'd like to start a vote for adding two new time API's to
>>>>>> ProcessorContext.
>>>>>>>>
>>>>>>>> Add currentSystemTimeMs and currentStreamTimeMs to
>> ProcessorContext
>>>>>>>> <
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks everyone for the initial feedback and thanks for your
>> time.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Rohit Deshpande <ro...@gmail.com>.
Hello all,
Based on the feedback of the pr <https://github.com/apache/kafka/pull/9744>
https://github.com/apache/kafka/pull/9744, there are following changes done
to the kip
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext>
.
*ProcessorContext#currentSystemTimeMs()*
It is expected that this method will return the internally cached system
timestamp from the Kafka Stream runtime. Thus, it may return a different
value compared to System.currentTimeMillis(). The cached system time
represents the time when we start processing / punctuating, and it would
not change throughout the process / punctuate. So this method will return
current system time (also called wall-clock time) known from kafka streams
runtime.
New methods to MockProcessorContext for testing purposes:
*MockProcessorContext#setRecordTimestamp*: set record timestamp
*MockProcessorContext#setCurrentSystemTimeMs:* set system timestamp
*MockProcessorContext#setCurrentStreamTimeMs*: set stream time
Deprecate method: MockProcessorContext#setTimestamp as it's name is
misleading and we are adding a new method
MockProcessorContext#setRecordTimestamp which does the same work.
Please let me know if you have any thoughts or concerns with this change.
Thanks,
Roohit
On Fri, Dec 4, 2020 at 7:31 PM Rohit Deshpande <ro...@gmail.com>
wrote:
> Hello all,
> I am closing the vote for this KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
>
> Summary of the KIP:
> Planning to add two new methods to ProcessorContext:
> 1. long currentSystemTimeMs() to fetch wall-clock time
> 2. long currentStreamTimeMs() to fetch maximum timestamp of any record yet
> processed by the task
>
> Thanks,
> Rohit
>
>
> On 2020/12/01 16:09:54, Bill Bejeck <bb...@gmail.com> wrote:
> > Sorry for jumping into this so late,
> >
> > Thanks for the KIP, I'm a +1 (binding)
> >
> > -Bill
> >
> > On Sun, Jul 26, 2020 at 11:06 AM John Roesler <jo...@vvcephei.org> wrote:
> >
> > > Thanks William,
> > >
> > > I’m +1 (binding)
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> > > > Thanks all, +1 (non-binding)
> > > >
> > > > Cheers,
> > > > Sophie
> > > >
> > > > On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io>
> wrote:
> > > >
> > > > > Thanks Will and Piotr,
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a vote for adding two new time API's to
> > > > > ProcessorContext.
> > > > > > >
> > > > > > > Add currentSystemTimeMs and currentStreamTimeMs to
> ProcessorContext
> > > > > > > <
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > > > > >
> > > > > > >
> > > > > > > Thanks everyone for the initial feedback and thanks for your
> time.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Rohit Deshpande <ro...@gmail.com>.
Hello all,
I am closing the vote for this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
Summary of the KIP:
Planning to add two new methods to ProcessorContext:
1. long currentSystemTimeMs() to fetch wall-clock time
2. long currentStreamTimeMs() to fetch maximum timestamp of any record yet processed by the task
Thanks,
Rohit
On 2020/12/01 16:09:54, Bill Bejeck <bb...@gmail.com> wrote:
> Sorry for jumping into this so late,
>
> Thanks for the KIP, I'm a +1 (binding)
>
> -Bill
>
> On Sun, Jul 26, 2020 at 11:06 AM John Roesler <jo...@vvcephei.org> wrote:
>
> > Thanks William,
> >
> > I’m +1 (binding)
> >
> > Thanks,
> > John
> >
> > On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> > > Thanks all, +1 (non-binding)
> > >
> > > Cheers,
> > > Sophie
> > >
> > > On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io> wrote:
> > >
> > > > Thanks Will and Piotr,
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > I'd like to start a vote for adding two new time API's to
> > > > ProcessorContext.
> > > > > >
> > > > > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > > > > <
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > > > >
> > > > > >
> > > > > > Thanks everyone for the initial feedback and thanks for your time.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Bill Bejeck <bb...@gmail.com>.
Sorry for jumping into this so late,
Thanks for the KIP, I'm a +1 (binding)
-Bill
On Sun, Jul 26, 2020 at 11:06 AM John Roesler <jo...@vvcephei.org> wrote:
> Thanks William,
>
> I’m +1 (binding)
>
> Thanks,
> John
>
> On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> > Thanks all, +1 (non-binding)
> >
> > Cheers,
> > Sophie
> >
> > On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io> wrote:
> >
> > > Thanks Will and Piotr,
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Bruno
> > >
> > > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a vote for adding two new time API's to
> > > ProcessorContext.
> > > > >
> > > > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > > >
> > > > >
> > > > > Thanks everyone for the initial feedback and thanks for your time.
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by John Roesler <jo...@vvcephei.org>.
Thanks William,
I’m +1 (binding)
Thanks,
John
On Fri, Jul 24, 2020, at 20:22, Sophie Blee-Goldman wrote:
> Thanks all, +1 (non-binding)
>
> Cheers,
> Sophie
>
> On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io> wrote:
>
> > Thanks Will and Piotr,
> >
> > +1 (non-binding)
> >
> > Best,
> > Bruno
> >
> > On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org> wrote:
> > >
> > > Thanks for the KIP.
> > >
> > > +1 (binding)
> > >
> > >
> > > -Matthias
> > >
> > > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > > Hi everyone,
> > > >
> > > > I'd like to start a vote for adding two new time API's to
> > ProcessorContext.
> > > >
> > > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > >
> > > >
> > > > Thanks everyone for the initial feedback and thanks for your time.
> > > >
> > >
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Sophie Blee-Goldman <so...@confluent.io>.
Thanks all, +1 (non-binding)
Cheers,
Sophie
On Wed, Jul 8, 2020 at 4:02 AM Bruno Cadonna <br...@confluent.io> wrote:
> Thanks Will and Piotr,
>
> +1 (non-binding)
>
> Best,
> Bruno
>
> On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org> wrote:
> >
> > Thanks for the KIP.
> >
> > +1 (binding)
> >
> >
> > -Matthias
> >
> > On 7/7/20 11:48 AM, William Bottrell wrote:
> > > Hi everyone,
> > >
> > > I'd like to start a vote for adding two new time API's to
> ProcessorContext.
> > >
> > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> >
> > >
> > > Thanks everyone for the initial feedback and thanks for your time.
> > >
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
Posted by Bruno Cadonna <br...@confluent.io>.
Thanks Will and Piotr,
+1 (non-binding)
Best,
Bruno
On Wed, Jul 8, 2020 at 8:12 AM Matthias J. Sax <mj...@apache.org> wrote:
>
> Thanks for the KIP.
>
> +1 (binding)
>
>
> -Matthias
>
> On 7/7/20 11:48 AM, William Bottrell wrote:
> > Hi everyone,
> >
> > I'd like to start a vote for adding two new time API's to ProcessorContext.
> >
> > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext>
> >
> > Thanks everyone for the initial feedback and thanks for your time.
> >
>
Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to
ProcessorContext
Posted by "Matthias J. Sax" <mj...@apache.org>.
Thanks for the KIP.
+1 (binding)
-Matthias
On 7/7/20 11:48 AM, William Bottrell wrote:
> Hi everyone,
>
> I'd like to start a vote for adding two new time API's to ProcessorContext.
>
> Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext>
>
> Thanks everyone for the initial feedback and thanks for your time.
>