You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Paolo Patierno <pp...@live.com> on 2018/02/01 08:49:50 UTC
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Hi Matthias,
just a question : what will be the timestamp "type" in the new message on the wire ?
Thanks,
Paolo.
________________________________
From: Matthias J. Sax <ma...@confluent.io>
Sent: Wednesday, January 31, 2018 2:06 AM
To: dev@kafka.apache.org
Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Hi,
I want to propose a new KIP for Kafka Streams that allows timestamp
manipulation at Processor API level.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-251%3A+Allow+timestamp+manipulation+in+Processor+API
Looking forward to your feedback.
-Matthias
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I updated the KIP to not include forwarding via child-index with the new
`To` class.
I also updated the PR accordingly.
It seems there are not further comments atm. I am going to start a VOTE
thread. Of course, you can always follow up on this thread.
-Matthias
On 2/7/18 2:10 PM, Guozhang Wang wrote:
> I intend to suggest we piggyback this change, just for connivence.
>
>
> Guozhang
>
> On Tue, Feb 6, 2018 at 7:25 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> I am open to remove the index API and also open to piggyback this change
>> on this KIP... Not sure if this was the intention of your comment? Or
>> should be have a separate KIP for this?
>>
>> Not sure, what other think about removing the index API?
>>
>>
>> -Matthias
>>
>> On 2/5/18 4:13 PM, Guozhang Wang wrote:
>>> This is not included in this KIP, but I'm wondering if we should still
>>> support forwarding to a child with index.
>>>
>>> Because 1) if users are using DSL, they would probably never use this API
>>> since the ordering of the child is hard to reason from a DSL anyways.
>>> 2) if users are using PAPI, they would just use the other
>>> overload with child name.
>>>
>>> In fact the index currently is only used at KStream.branch internally,
>> and
>>> maybe we can just keep a map from predicate to child name inside Filter
>>> operator, so that we do not use it at all. Just as a hindsight, I feel
>> this
>>> overloading API is not useful to users at all.
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Thu, Feb 1, 2018 at 5:05 PM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks.
>>>>
>>>> I updated the KIP accordingly and started work on the PR to see if this
>>>> `To` interface work nicely.
>>>>
>>>> -Matthias
>>>>
>>>> On 2/1/18 4:00 PM, Ted Yu wrote:
>>>>> Yeah.
>>>>> Cleaner in this formation.
>>>>>
>>>>> On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
>>>>>
>>>>>> `To` works for me.
>>>>>>
>>>>>> Thanks,
>>>>>> Bill
>>>>>>
>>>>>> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> @Paolo:
>>>>>>>
>>>>>>> The timestamp will be used to set the message/record metadata
>> timestamp
>>>>>>> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
>>>>>>>
>>>>>>> @Bill,Ted:
>>>>>>>
>>>>>>> Might be a good idea. I was thinking about the name, and came up with
>>>>>> `To`:
>>>>>>>
>>>>>>>> context.forward(key, value, To.child("processorX").
>> withTimestamp(5));
>>>>>>>> context.forward(key, value, To.child(1).withTimestamp(10));
>>>>>>>
>>>>>>> Without specifying the downstream child processor:
>>>>>>>
>>>>>>>> context.forward(key, value, To.all().withTimestamp(5));
>>>>>>>
>>>>>>> WDYT?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 2/1/18 8:45 AM, Ted Yu wrote:
>>>>>>>> I like Bill's idea (pending a better name for the Forwarded).
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Matthias,
>>>>>>>>>
>>>>>>>>> Thanks for the KIP!
>>>>>>>>>
>>>>>>>>> Could we consider taking an approach similar to what was done in
>>>>>> KIP-182
>>>>>>>>> with regards to overloading?
>>>>>>>>>
>>>>>>>>> Meaning we could add a "Forwarded" object (horrible name I know)
>> with
>>>>>>>>> methods withTimestamp, withChildName, and withChildIndex. To handle
>>>>>> the
>>>>>>>>> case when both a child-name and child-index is provided we could
>>>> throw
>>>>>>> an
>>>>>>>>> exception.
>>>>>>>>>
>>>>>>>>> Then we could reduce the overloaded {{forward}} methods from 6 to
>> 2.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bill
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <ppatierno@live.com
>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Matthias,
>>>>>>>>>>
>>>>>>>>>> just a question : what will be the timestamp "type" in the new
>>>>>> message
>>>>>>> on
>>>>>>>>>> the wire ?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Paolo.
>>>>>>>>>> ________________________________
>>>>>>>>>> From: Matthias J. Sax <ma...@confluent.io>
>>>>>>>>>> Sent: Wednesday, January 31, 2018 2:06 AM
>>>>>>>>>> To: dev@kafka.apache.org
>>>>>>>>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in
>>>> Processor
>>>>>>> API
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I want to propose a new KIP for Kafka Streams that allows
>> timestamp
>>>>>>>>>> manipulation at Processor API level.
>>>>>>>>>>
>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>>>>>>>>>>
>>>>>>>>>> Looking forward to your feedback.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Guozhang Wang <wa...@gmail.com>.
I intend to suggest we piggyback this change, just for connivence.
Guozhang
On Tue, Feb 6, 2018 at 7:25 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> I am open to remove the index API and also open to piggyback this change
> on this KIP... Not sure if this was the intention of your comment? Or
> should be have a separate KIP for this?
>
> Not sure, what other think about removing the index API?
>
>
> -Matthias
>
> On 2/5/18 4:13 PM, Guozhang Wang wrote:
> > This is not included in this KIP, but I'm wondering if we should still
> > support forwarding to a child with index.
> >
> > Because 1) if users are using DSL, they would probably never use this API
> > since the ordering of the child is hard to reason from a DSL anyways.
> > 2) if users are using PAPI, they would just use the other
> > overload with child name.
> >
> > In fact the index currently is only used at KStream.branch internally,
> and
> > maybe we can just keep a map from predicate to child name inside Filter
> > operator, so that we do not use it at all. Just as a hindsight, I feel
> this
> > overloading API is not useful to users at all.
> >
> >
> > Guozhang
> >
> >
> > On Thu, Feb 1, 2018 at 5:05 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks.
> >>
> >> I updated the KIP accordingly and started work on the PR to see if this
> >> `To` interface work nicely.
> >>
> >> -Matthias
> >>
> >> On 2/1/18 4:00 PM, Ted Yu wrote:
> >>> Yeah.
> >>> Cleaner in this formation.
> >>>
> >>> On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
> >>>
> >>>> `To` works for me.
> >>>>
> >>>> Thanks,
> >>>> Bill
> >>>>
> >>>> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <
> matthias@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> @Paolo:
> >>>>>
> >>>>> The timestamp will be used to set the message/record metadata
> timestamp
> >>>>> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
> >>>>>
> >>>>> @Bill,Ted:
> >>>>>
> >>>>> Might be a good idea. I was thinking about the name, and came up with
> >>>> `To`:
> >>>>>
> >>>>>> context.forward(key, value, To.child("processorX").
> withTimestamp(5));
> >>>>>> context.forward(key, value, To.child(1).withTimestamp(10));
> >>>>>
> >>>>> Without specifying the downstream child processor:
> >>>>>
> >>>>>> context.forward(key, value, To.all().withTimestamp(5));
> >>>>>
> >>>>> WDYT?
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 2/1/18 8:45 AM, Ted Yu wrote:
> >>>>>> I like Bill's idea (pending a better name for the Forwarded).
> >>>>>>
> >>>>>> Cheers
> >>>>>>
> >>>>>> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> Hi Matthias,
> >>>>>>>
> >>>>>>> Thanks for the KIP!
> >>>>>>>
> >>>>>>> Could we consider taking an approach similar to what was done in
> >>>> KIP-182
> >>>>>>> with regards to overloading?
> >>>>>>>
> >>>>>>> Meaning we could add a "Forwarded" object (horrible name I know)
> with
> >>>>>>> methods withTimestamp, withChildName, and withChildIndex. To handle
> >>>> the
> >>>>>>> case when both a child-name and child-index is provided we could
> >> throw
> >>>>> an
> >>>>>>> exception.
> >>>>>>>
> >>>>>>> Then we could reduce the overloaded {{forward}} methods from 6 to
> 2.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Bill
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <ppatierno@live.com
> >
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Matthias,
> >>>>>>>>
> >>>>>>>> just a question : what will be the timestamp "type" in the new
> >>>> message
> >>>>> on
> >>>>>>>> the wire ?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Paolo.
> >>>>>>>> ________________________________
> >>>>>>>> From: Matthias J. Sax <ma...@confluent.io>
> >>>>>>>> Sent: Wednesday, January 31, 2018 2:06 AM
> >>>>>>>> To: dev@kafka.apache.org
> >>>>>>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in
> >> Processor
> >>>>> API
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I want to propose a new KIP for Kafka Streams that allows
> timestamp
> >>>>>>>> manipulation at Processor API level.
> >>>>>>>>
> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
> >>>>>>>>
> >>>>>>>> Looking forward to your feedback.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
> >
>
>
--
-- Guozhang
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I am open to remove the index API and also open to piggyback this change
on this KIP... Not sure if this was the intention of your comment? Or
should be have a separate KIP for this?
Not sure, what other think about removing the index API?
-Matthias
On 2/5/18 4:13 PM, Guozhang Wang wrote:
> This is not included in this KIP, but I'm wondering if we should still
> support forwarding to a child with index.
>
> Because 1) if users are using DSL, they would probably never use this API
> since the ordering of the child is hard to reason from a DSL anyways.
> 2) if users are using PAPI, they would just use the other
> overload with child name.
>
> In fact the index currently is only used at KStream.branch internally, and
> maybe we can just keep a map from predicate to child name inside Filter
> operator, so that we do not use it at all. Just as a hindsight, I feel this
> overloading API is not useful to users at all.
>
>
> Guozhang
>
>
> On Thu, Feb 1, 2018 at 5:05 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks.
>>
>> I updated the KIP accordingly and started work on the PR to see if this
>> `To` interface work nicely.
>>
>> -Matthias
>>
>> On 2/1/18 4:00 PM, Ted Yu wrote:
>>> Yeah.
>>> Cleaner in this formation.
>>>
>>> On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
>>>
>>>> `To` works for me.
>>>>
>>>> Thanks,
>>>> Bill
>>>>
>>>> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> @Paolo:
>>>>>
>>>>> The timestamp will be used to set the message/record metadata timestamp
>>>>> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
>>>>>
>>>>> @Bill,Ted:
>>>>>
>>>>> Might be a good idea. I was thinking about the name, and came up with
>>>> `To`:
>>>>>
>>>>>> context.forward(key, value, To.child("processorX").withTimestamp(5));
>>>>>> context.forward(key, value, To.child(1).withTimestamp(10));
>>>>>
>>>>> Without specifying the downstream child processor:
>>>>>
>>>>>> context.forward(key, value, To.all().withTimestamp(5));
>>>>>
>>>>> WDYT?
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 2/1/18 8:45 AM, Ted Yu wrote:
>>>>>> I like Bill's idea (pending a better name for the Forwarded).
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com>
>> wrote:
>>>>>>
>>>>>>> Hi Matthias,
>>>>>>>
>>>>>>> Thanks for the KIP!
>>>>>>>
>>>>>>> Could we consider taking an approach similar to what was done in
>>>> KIP-182
>>>>>>> with regards to overloading?
>>>>>>>
>>>>>>> Meaning we could add a "Forwarded" object (horrible name I know) with
>>>>>>> methods withTimestamp, withChildName, and withChildIndex. To handle
>>>> the
>>>>>>> case when both a child-name and child-index is provided we could
>> throw
>>>>> an
>>>>>>> exception.
>>>>>>>
>>>>>>> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bill
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com>
>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Matthias,
>>>>>>>>
>>>>>>>> just a question : what will be the timestamp "type" in the new
>>>> message
>>>>> on
>>>>>>>> the wire ?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Paolo.
>>>>>>>> ________________________________
>>>>>>>> From: Matthias J. Sax <ma...@confluent.io>
>>>>>>>> Sent: Wednesday, January 31, 2018 2:06 AM
>>>>>>>> To: dev@kafka.apache.org
>>>>>>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in
>> Processor
>>>>> API
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I want to propose a new KIP for Kafka Streams that allows timestamp
>>>>>>>> manipulation at Processor API level.
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>>>>>>>>
>>>>>>>> Looking forward to your feedback.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Guozhang Wang <wa...@gmail.com>.
This is not included in this KIP, but I'm wondering if we should still
support forwarding to a child with index.
Because 1) if users are using DSL, they would probably never use this API
since the ordering of the child is hard to reason from a DSL anyways.
2) if users are using PAPI, they would just use the other
overload with child name.
In fact the index currently is only used at KStream.branch internally, and
maybe we can just keep a map from predicate to child name inside Filter
operator, so that we do not use it at all. Just as a hindsight, I feel this
overloading API is not useful to users at all.
Guozhang
On Thu, Feb 1, 2018 at 5:05 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks.
>
> I updated the KIP accordingly and started work on the PR to see if this
> `To` interface work nicely.
>
> -Matthias
>
> On 2/1/18 4:00 PM, Ted Yu wrote:
> > Yeah.
> > Cleaner in this formation.
> >
> > On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
> >
> >> `To` works for me.
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <ma...@confluent.io>
> >> wrote:
> >>
> >>> @Paolo:
> >>>
> >>> The timestamp will be used to set the message/record metadata timestamp
> >>> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
> >>>
> >>> @Bill,Ted:
> >>>
> >>> Might be a good idea. I was thinking about the name, and came up with
> >> `To`:
> >>>
> >>>> context.forward(key, value, To.child("processorX").withTimestamp(5));
> >>>> context.forward(key, value, To.child(1).withTimestamp(10));
> >>>
> >>> Without specifying the downstream child processor:
> >>>
> >>>> context.forward(key, value, To.all().withTimestamp(5));
> >>>
> >>> WDYT?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 2/1/18 8:45 AM, Ted Yu wrote:
> >>>> I like Bill's idea (pending a better name for the Forwarded).
> >>>>
> >>>> Cheers
> >>>>
> >>>> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com>
> wrote:
> >>>>
> >>>>> Hi Matthias,
> >>>>>
> >>>>> Thanks for the KIP!
> >>>>>
> >>>>> Could we consider taking an approach similar to what was done in
> >> KIP-182
> >>>>> with regards to overloading?
> >>>>>
> >>>>> Meaning we could add a "Forwarded" object (horrible name I know) with
> >>>>> methods withTimestamp, withChildName, and withChildIndex. To handle
> >> the
> >>>>> case when both a child-name and child-index is provided we could
> throw
> >>> an
> >>>>> exception.
> >>>>>
> >>>>> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
> >>>>>
> >>>>> Thanks,
> >>>>> Bill
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi Matthias,
> >>>>>>
> >>>>>> just a question : what will be the timestamp "type" in the new
> >> message
> >>> on
> >>>>>> the wire ?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Paolo.
> >>>>>> ________________________________
> >>>>>> From: Matthias J. Sax <ma...@confluent.io>
> >>>>>> Sent: Wednesday, January 31, 2018 2:06 AM
> >>>>>> To: dev@kafka.apache.org
> >>>>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in
> Processor
> >>> API
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I want to propose a new KIP for Kafka Streams that allows timestamp
> >>>>>> manipulation at Processor API level.
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
> >>>>>>
> >>>>>> Looking forward to your feedback.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>
--
-- Guozhang
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks.
I updated the KIP accordingly and started work on the PR to see if this
`To` interface work nicely.
-Matthias
On 2/1/18 4:00 PM, Ted Yu wrote:
> Yeah.
> Cleaner in this formation.
>
> On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
>
>> `To` works for me.
>>
>> Thanks,
>> Bill
>>
>> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> @Paolo:
>>>
>>> The timestamp will be used to set the message/record metadata timestamp
>>> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
>>>
>>> @Bill,Ted:
>>>
>>> Might be a good idea. I was thinking about the name, and came up with
>> `To`:
>>>
>>>> context.forward(key, value, To.child("processorX").withTimestamp(5));
>>>> context.forward(key, value, To.child(1).withTimestamp(10));
>>>
>>> Without specifying the downstream child processor:
>>>
>>>> context.forward(key, value, To.all().withTimestamp(5));
>>>
>>> WDYT?
>>>
>>>
>>> -Matthias
>>>
>>> On 2/1/18 8:45 AM, Ted Yu wrote:
>>>> I like Bill's idea (pending a better name for the Forwarded).
>>>>
>>>> Cheers
>>>>
>>>> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com> wrote:
>>>>
>>>>> Hi Matthias,
>>>>>
>>>>> Thanks for the KIP!
>>>>>
>>>>> Could we consider taking an approach similar to what was done in
>> KIP-182
>>>>> with regards to overloading?
>>>>>
>>>>> Meaning we could add a "Forwarded" object (horrible name I know) with
>>>>> methods withTimestamp, withChildName, and withChildIndex. To handle
>> the
>>>>> case when both a child-name and child-index is provided we could throw
>>> an
>>>>> exception.
>>>>>
>>>>> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
>>>>>
>>>>> Thanks,
>>>>> Bill
>>>>>
>>>>>
>>>>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com>
>>> wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>> just a question : what will be the timestamp "type" in the new
>> message
>>> on
>>>>>> the wire ?
>>>>>>
>>>>>> Thanks,
>>>>>> Paolo.
>>>>>> ________________________________
>>>>>> From: Matthias J. Sax <ma...@confluent.io>
>>>>>> Sent: Wednesday, January 31, 2018 2:06 AM
>>>>>> To: dev@kafka.apache.org
>>>>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor
>>> API
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want to propose a new KIP for Kafka Streams that allows timestamp
>>>>>> manipulation at Processor API level.
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>>>>>>
>>>>>> Looking forward to your feedback.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Ted Yu <yu...@gmail.com>.
Yeah.
Cleaner in this formation.
On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck <bb...@gmail.com> wrote:
> `To` works for me.
>
> Thanks,
> Bill
>
> On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > @Paolo:
> >
> > The timestamp will be used to set the message/record metadata timestamp
> > on `Producer.send(new ProducerRecord(...,timestamp,...))`.
> >
> > @Bill,Ted:
> >
> > Might be a good idea. I was thinking about the name, and came up with
> `To`:
> >
> > > context.forward(key, value, To.child("processorX").withTimestamp(5));
> > > context.forward(key, value, To.child(1).withTimestamp(10));
> >
> > Without specifying the downstream child processor:
> >
> > > context.forward(key, value, To.all().withTimestamp(5));
> >
> > WDYT?
> >
> >
> > -Matthias
> >
> > On 2/1/18 8:45 AM, Ted Yu wrote:
> > > I like Bill's idea (pending a better name for the Forwarded).
> > >
> > > Cheers
> > >
> > > On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com> wrote:
> > >
> > >> Hi Matthias,
> > >>
> > >> Thanks for the KIP!
> > >>
> > >> Could we consider taking an approach similar to what was done in
> KIP-182
> > >> with regards to overloading?
> > >>
> > >> Meaning we could add a "Forwarded" object (horrible name I know) with
> > >> methods withTimestamp, withChildName, and withChildIndex. To handle
> the
> > >> case when both a child-name and child-index is provided we could throw
> > an
> > >> exception.
> > >>
> > >> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
> > >>
> > >> Thanks,
> > >> Bill
> > >>
> > >>
> > >> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com>
> > wrote:
> > >>
> > >>> Hi Matthias,
> > >>>
> > >>> just a question : what will be the timestamp "type" in the new
> message
> > on
> > >>> the wire ?
> > >>>
> > >>> Thanks,
> > >>> Paolo.
> > >>> ________________________________
> > >>> From: Matthias J. Sax <ma...@confluent.io>
> > >>> Sent: Wednesday, January 31, 2018 2:06 AM
> > >>> To: dev@kafka.apache.org
> > >>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor
> > API
> > >>>
> > >>> Hi,
> > >>>
> > >>> I want to propose a new KIP for Kafka Streams that allows timestamp
> > >>> manipulation at Processor API level.
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
> > >>>
> > >>> Looking forward to your feedback.
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Bill Bejeck <bb...@gmail.com>.
`To` works for me.
Thanks,
Bill
On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> @Paolo:
>
> The timestamp will be used to set the message/record metadata timestamp
> on `Producer.send(new ProducerRecord(...,timestamp,...))`.
>
> @Bill,Ted:
>
> Might be a good idea. I was thinking about the name, and came up with `To`:
>
> > context.forward(key, value, To.child("processorX").withTimestamp(5));
> > context.forward(key, value, To.child(1).withTimestamp(10));
>
> Without specifying the downstream child processor:
>
> > context.forward(key, value, To.all().withTimestamp(5));
>
> WDYT?
>
>
> -Matthias
>
> On 2/1/18 8:45 AM, Ted Yu wrote:
> > I like Bill's idea (pending a better name for the Forwarded).
> >
> > Cheers
> >
> > On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com> wrote:
> >
> >> Hi Matthias,
> >>
> >> Thanks for the KIP!
> >>
> >> Could we consider taking an approach similar to what was done in KIP-182
> >> with regards to overloading?
> >>
> >> Meaning we could add a "Forwarded" object (horrible name I know) with
> >> methods withTimestamp, withChildName, and withChildIndex. To handle the
> >> case when both a child-name and child-index is provided we could throw
> an
> >> exception.
> >>
> >> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
> >>
> >> Thanks,
> >> Bill
> >>
> >>
> >> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com>
> wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> just a question : what will be the timestamp "type" in the new message
> on
> >>> the wire ?
> >>>
> >>> Thanks,
> >>> Paolo.
> >>> ________________________________
> >>> From: Matthias J. Sax <ma...@confluent.io>
> >>> Sent: Wednesday, January 31, 2018 2:06 AM
> >>> To: dev@kafka.apache.org
> >>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor
> API
> >>>
> >>> Hi,
> >>>
> >>> I want to propose a new KIP for Kafka Streams that allows timestamp
> >>> manipulation at Processor API level.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
> >>>
> >>> Looking forward to your feedback.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>
> >
>
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by "Matthias J. Sax" <ma...@confluent.io>.
@Paolo:
The timestamp will be used to set the message/record metadata timestamp
on `Producer.send(new ProducerRecord(...,timestamp,...))`.
@Bill,Ted:
Might be a good idea. I was thinking about the name, and came up with `To`:
> context.forward(key, value, To.child("processorX").withTimestamp(5));
> context.forward(key, value, To.child(1).withTimestamp(10));
Without specifying the downstream child processor:
> context.forward(key, value, To.all().withTimestamp(5));
WDYT?
-Matthias
On 2/1/18 8:45 AM, Ted Yu wrote:
> I like Bill's idea (pending a better name for the Forwarded).
>
> Cheers
>
> On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com> wrote:
>
>> Hi Matthias,
>>
>> Thanks for the KIP!
>>
>> Could we consider taking an approach similar to what was done in KIP-182
>> with regards to overloading?
>>
>> Meaning we could add a "Forwarded" object (horrible name I know) with
>> methods withTimestamp, withChildName, and withChildIndex. To handle the
>> case when both a child-name and child-index is provided we could throw an
>> exception.
>>
>> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
>>
>> Thanks,
>> Bill
>>
>>
>> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com> wrote:
>>
>>> Hi Matthias,
>>>
>>> just a question : what will be the timestamp "type" in the new message on
>>> the wire ?
>>>
>>> Thanks,
>>> Paolo.
>>> ________________________________
>>> From: Matthias J. Sax <ma...@confluent.io>
>>> Sent: Wednesday, January 31, 2018 2:06 AM
>>> To: dev@kafka.apache.org
>>> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
>>>
>>> Hi,
>>>
>>> I want to propose a new KIP for Kafka Streams that allows timestamp
>>> manipulation at Processor API level.
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>>>
>>> Looking forward to your feedback.
>>>
>>>
>>> -Matthias
>>>
>>>
>>
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Ted Yu <yu...@gmail.com>.
I like Bill's idea (pending a better name for the Forwarded).
Cheers
On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck <bb...@gmail.com> wrote:
> Hi Matthias,
>
> Thanks for the KIP!
>
> Could we consider taking an approach similar to what was done in KIP-182
> with regards to overloading?
>
> Meaning we could add a "Forwarded" object (horrible name I know) with
> methods withTimestamp, withChildName, and withChildIndex. To handle the
> case when both a child-name and child-index is provided we could throw an
> exception.
>
> Then we could reduce the overloaded {{forward}} methods from 6 to 2.
>
> Thanks,
> Bill
>
>
> On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com> wrote:
>
> > Hi Matthias,
> >
> > just a question : what will be the timestamp "type" in the new message on
> > the wire ?
> >
> > Thanks,
> > Paolo.
> > ________________________________
> > From: Matthias J. Sax <ma...@confluent.io>
> > Sent: Wednesday, January 31, 2018 2:06 AM
> > To: dev@kafka.apache.org
> > Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
> >
> > Hi,
> >
> > I want to propose a new KIP for Kafka Streams that allows timestamp
> > manipulation at Processor API level.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 251%3A+Allow+timestamp+manipulation+in+Processor+API
> >
> > Looking forward to your feedback.
> >
> >
> > -Matthias
> >
> >
>
Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
Posted by Bill Bejeck <bb...@gmail.com>.
Hi Matthias,
Thanks for the KIP!
Could we consider taking an approach similar to what was done in KIP-182
with regards to overloading?
Meaning we could add a "Forwarded" object (horrible name I know) with
methods withTimestamp, withChildName, and withChildIndex. To handle the
case when both a child-name and child-index is provided we could throw an
exception.
Then we could reduce the overloaded {{forward}} methods from 6 to 2.
Thanks,
Bill
On Thu, Feb 1, 2018 at 3:49 AM, Paolo Patierno <pp...@live.com> wrote:
> Hi Matthias,
>
> just a question : what will be the timestamp "type" in the new message on
> the wire ?
>
> Thanks,
> Paolo.
> ________________________________
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: Wednesday, January 31, 2018 2:06 AM
> To: dev@kafka.apache.org
> Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API
>
> Hi,
>
> I want to propose a new KIP for Kafka Streams that allows timestamp
> manipulation at Processor API level.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 251%3A+Allow+timestamp+manipulation+in+Processor+API
>
> Looking forward to your feedback.
>
>
> -Matthias
>
>