You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2018/09/10 17:54:14 UTC
Re: [VOTE] KIP-349 Priorities for Source Topics
Hmm. My understanding is that streams doesn't need anything like this since streams pauses topics when it doesn't need more data from them. (Matthias, can you confirm?)
best,
Colin
On Mon, Aug 20, 2018, at 06:01, Thomas Becker wrote:
> I agree with Jan. A strategy interface for choosing processing order is
> nice, and would hopefully be a step towards getting this in streams.
>
> -Tommy
>
> On Mon, 2018-08-20 at 12:52 +0200, Jan Filipiak wrote:
>
> On 20.08.2018 00:19, Matthias J. Sax wrote:
>
> @Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from
>
> committers. If you close the vote before that, the KIP would not be
>
> accepted. Note that committers need to pay attention to a lot of KIPs
>
> and it can take a while until people can look into it. Thanks for your
>
> understanding.
>
>
> @Jan: Can you give a little bit more context on your concerns? It's
>
> unclear why you mean atm.
>
> Just saying that we should peek at the Samza approach, it's a much more
>
> powerful abstraction. We can ship a default MessageChooser
>
> that looks at the topics priority.
>
> @Adam: anyone can vote :)
>
>
>
>
> -Matthias
>
>
> On 8/19/18 9:58 AM, Adam Bellemare wrote:
>
> While I am not sure if I can or can’t vote, my question re: Jan’s
> comment is, “should we be implementing it as Samza does?”
>
>
> I am not familiar with the drawbacks of the current approach vs how
> samza does it.
>
>
> On Aug 18, 2018, at 5:06 PM,
> nick@afshartous.com<ma...@afshartous.com> wrote:
>
>
>
> I only saw one vote on KIP-349, just checking to see if anyone else
> would like to vote before closing this out.
>
> --
>
> Nick
>
>
>
> On Aug 13, 2018, at 9:19 PM,
> nick@afshartous.com<ma...@afshartous.com> wrote:
>
>
>
> Hi All,
>
>
> Calling for a vote on KIP-349
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics
>
>
> --
>
> Nick
>
>
>
>
>
>
>
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review,
> copying, or distribution of this email (or any attachments) by others is
> prohibited. If you are not the intended recipient, please contact the
> sender immediately and permanently delete this email and any
> attachments. No employee or agent of TiVo Inc. is authorized to conclude
> any binding agreement on behalf of TiVo Inc. by email. Binding
> agreements with TiVo Inc. may only be made by a signed written
> agreement.
Re: [VOTE] KIP-349 Priorities for Source Topics
Posted by "Matthias J. Sax" <ma...@confluent.io>.
That sound correct, Colin.
At runtime (we just merged an improvement this week, cf KIP-353), Kafka
Streams synchronizes different topics based on record timestamps.
Records are buffered internally before processed and we `pause()`
partitions for which the number of records in the buffer exceeds a
configurable threshold (`buffered.records.per.partition` parameter).
Timestamp based synchronization (ie, message choosing) is essential for
DSL semantics. A custom MessageChosser might break DSL operator semantics.
Having said this, there might be use cases for with timestamps based
synchronization is not desired. However, I would assume that this would
be a Processor API level feature, not a DSL level feature.
Hence, offering something similar to MessageChooser interface at
Processor API level that is leveraged at runtime might make sense. For
this case, the DSL would plug-in its timestamp based synchronization
strategy. Take this with a grain of salt though. I have not thought this
through and it might actually not be possible to express the needed
timestamp synchronization with a MessageChooser interface.
-Matthias
On 9/10/18 10:54 AM, Colin McCabe wrote:
> Hmm. My understanding is that streams doesn't need anything like this since streams pauses topics when it doesn't need more data from them. (Matthias, can you confirm?)
>
> best,
> Colin
>
>
> On Mon, Aug 20, 2018, at 06:01, Thomas Becker wrote:
>> I agree with Jan. A strategy interface for choosing processing order is
>> nice, and would hopefully be a step towards getting this in streams.
>>
>> -Tommy
>>
>> On Mon, 2018-08-20 at 12:52 +0200, Jan Filipiak wrote:
>>
>> On 20.08.2018 00:19, Matthias J. Sax wrote:
>>
>> @Nick: A KIP is only accepted if it got 3 binding votes, ie, votes from
>>
>> committers. If you close the vote before that, the KIP would not be
>>
>> accepted. Note that committers need to pay attention to a lot of KIPs
>>
>> and it can take a while until people can look into it. Thanks for your
>>
>> understanding.
>>
>>
>> @Jan: Can you give a little bit more context on your concerns? It's
>>
>> unclear why you mean atm.
>>
>> Just saying that we should peek at the Samza approach, it's a much more
>>
>> powerful abstraction. We can ship a default MessageChooser
>>
>> that looks at the topics priority.
>>
>> @Adam: anyone can vote :)
>>
>>
>>
>>
>> -Matthias
>>
>>
>> On 8/19/18 9:58 AM, Adam Bellemare wrote:
>>
>> While I am not sure if I can or can’t vote, my question re: Jan’s
>> comment is, “should we be implementing it as Samza does?”
>>
>>
>> I am not familiar with the drawbacks of the current approach vs how
>> samza does it.
>>
>>
>> On Aug 18, 2018, at 5:06 PM,
>> nick@afshartous.com<ma...@afshartous.com> wrote:
>>
>>
>>
>> I only saw one vote on KIP-349, just checking to see if anyone else
>> would like to vote before closing this out.
>>
>> --
>>
>> Nick
>>
>>
>>
>> On Aug 13, 2018, at 9:19 PM,
>> nick@afshartous.com<ma...@afshartous.com> wrote:
>>
>>
>>
>> Hi All,
>>
>>
>> Calling for a vote on KIP-349
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics
>>
>>
>> --
>>
>> Nick
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> This email and any attachments may contain confidential and privileged
>> material for the sole use of the intended recipient. Any review,
>> copying, or distribution of this email (or any attachments) by others is
>> prohibited. If you are not the intended recipient, please contact the
>> sender immediately and permanently delete this email and any
>> attachments. No employee or agent of TiVo Inc. is authorized to conclude
>> any binding agreement on behalf of TiVo Inc. by email. Binding
>> agreements with TiVo Inc. may only be made by a signed written
>> agreement.