You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matthias J. Sax" <ma...@confluent.io> on 2017/10/13 17:39:35 UTC
Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest
update timestamp per key
I wanted to follow up with this KIP.
One important thing that is missing in this KIP, is the question where
the TS is coming from? ATM, timestamps are not stored in the state
stores -- it's a plain <key,value> format.
Thus, if we want to enable this feature, we need to change the data
format of the stores. This also required an upgrade path for
applications to migrate from the old format to the new format.
-Matthias
On 6/27/17 2:16 PM, Jeyhun Karimov wrote:
> Hi Michal,
>
>
> Thanks a lot for your comment. I fixed the document.
>
> Cheers,
> Jeyhun
>
> On Sat, Jun 24, 2017 at 6:49 PM Michal Borowiecki
> <michal.borowiecki@openbet.com <ma...@openbet.com>>
> wrote:
>
> Hi Jeyhun,
>
> Could the proposed KeyContext.keyTs() be made more descriptive?
>
> e.g. lastUpdated() or similar? So that users don't have to read the
> docs to know it isn't the creation timestamp for instance.
>
> Cheers,
> Michał
>
>
> On 04/06/17 01:24, Jeyhun Karimov wrote:
>> Hi Matthias,
>>
>> Thanks for comments.
>>
>> - why do you only consider get() and not range() and all() ?
>>
>>
>> The corresponding jira concentrates on single key lookups. Moreover, I
>> could not find a use-case to include range queries to return records with
>> timestamp. However, theoritically we can include range() and all() as well.
>>
>> - we cannot have a second get() (this would be ambiguous) but need
>>> another name like getWithTs() (or something better)
>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>> method returning KeyContext not be sufficient?
>>
>> Thanks for correction, this is my bad.
>>
>> - for backward compatibility, we will also need a new interface and
>>> cannot just extend the existing one
>>
>> I will correct the KIP accordingly.
>>
>> Thanks,
>> Jeyhun
>>
>> On Fri, Jun 2, 2017 at 7:36 AM, Matthias J. Sax <ma...@confluent.io> <ma...@confluent.io>
>> wrote:
>>
>>> Thanks for the KIP Jeyhun.
>>>
>>> Some comments:
>>> - why do you only consider get() and not range() and all() ?
>>> - we cannot have a second get() (this would be ambiguous) but need
>>> another name like getWithTs() (or something better)
>>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>> method returning KeyContext not be sufficient?
>>> - for backward compatibility, we will also need a new interface and
>>> cannot just extend the existing one
>>>
>>>
>>>
>>> -Matthias
>>>
>>> On 5/29/17 4:55 PM, Jeyhun Karimov wrote:
>>>> Dear community,
>>>>
>>>> I want to share KIP-165 [1] based on issue KAFKA-4304 [2].
>>>> I would like to get your comments.
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 165%3A+Extend+Interactive+Queries+for+return+latest+
>>> update+timestamp+per+key
>>>> [2] https://issues.apache.org/jira/browse/KAFKA-4304
>>>>
>>>> Cheers,
>>>> Jeyhun
>>>>
>>>
>
> --
> <http://www.openbet.com/> Michal Borowiecki
> Senior Software Engineer L4
> T: +44 208 742 1600 <tel:+44%2020%208742%201600>
>
>
> +44 203 249 8448 <tel:+44%2020%203249%208448>
>
>
>
> E: michal.borowiecki@openbet.com
> <ma...@openbet.com>
> W: www.openbet.com <http://www.openbet.com/>
>
>
> OpenBet Ltd
>
> Chiswick Park Building 9
>
> 566 Chiswick High Rd
>
> London
>
> W4 5XT
>
> UK
>
>
> <https://www.openbet.com/email_promo>
>
> This message is confidential and intended only for the addressee. If
> you have received this message in error, please immediately notify
> the postmaster@openbet.com <ma...@openbet.com> and
> delete it from your system as well as any copies. The content of
> e-mails as well as traffic data may be monitored by OpenBet for
> employment and security purposes. To protect the environment please
> do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
> 5XT, United Kingdom. A company registered in England and Wales.
> Registered no. 3134634. VAT no. GB927523612
>
> --
> -Cheers
>
> Jeyhun
Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest
update timestamp per key
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi,
This issue seems to be resolved as part of KIP-258. Thus, I close this
discussion and mark this KIP as discarded.
Let me know if there is any objection. Thanks.
-Matthias
On 12/4/17 4:06 PM, Matthias J. Sax wrote:
> Jeyhun,
>
> is there any update regarding this KIP?
>
>
> -Matthias
>
> On 10/13/17 10:39 AM, Matthias J. Sax wrote:
>> I wanted to follow up with this KIP.
>>
>> One important thing that is missing in this KIP, is the question where
>> the TS is coming from? ATM, timestamps are not stored in the state
>> stores -- it's a plain <key,value> format.
>>
>> Thus, if we want to enable this feature, we need to change the data
>> format of the stores. This also required an upgrade path for
>> applications to migrate from the old format to the new format.
>>
>>
>> -Matthias
>>
>> On 6/27/17 2:16 PM, Jeyhun Karimov wrote:
>>> Hi Michal,
>>>
>>>
>>> Thanks a lot for your comment. I fixed the document.
>>>
>>> Cheers,
>>> Jeyhun
>>>
>>> On Sat, Jun 24, 2017 at 6:49 PM Michal Borowiecki
>>> <michal.borowiecki@openbet.com <ma...@openbet.com>>
>>> wrote:
>>>
>>> Hi Jeyhun,
>>>
>>> Could the proposed KeyContext.keyTs() be made more descriptive?
>>>
>>> e.g. lastUpdated() or similar? So that users don't have to read the
>>> docs to know it isn't the creation timestamp for instance.
>>>
>>> Cheers,
>>> Michał
>>>
>>>
>>> On 04/06/17 01:24, Jeyhun Karimov wrote:
>>>> Hi Matthias,
>>>>
>>>> Thanks for comments.
>>>>
>>>> - why do you only consider get() and not range() and all() ?
>>>>
>>>>
>>>> The corresponding jira concentrates on single key lookups. Moreover, I
>>>> could not find a use-case to include range queries to return records with
>>>> timestamp. However, theoritically we can include range() and all() as well.
>>>>
>>>> - we cannot have a second get() (this would be ambiguous) but need
>>>>> another name like getWithTs() (or something better)
>>>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>>>> method returning KeyContext not be sufficient?
>>>>
>>>> Thanks for correction, this is my bad.
>>>>
>>>> - for backward compatibility, we will also need a new interface and
>>>>> cannot just extend the existing one
>>>>
>>>> I will correct the KIP accordingly.
>>>>
>>>> Thanks,
>>>> Jeyhun
>>>>
>>>> On Fri, Jun 2, 2017 at 7:36 AM, Matthias J. Sax <ma...@confluent.io> <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> Thanks for the KIP Jeyhun.
>>>>>
>>>>> Some comments:
>>>>> - why do you only consider get() and not range() and all() ?
>>>>> - we cannot have a second get() (this would be ambiguous) but need
>>>>> another name like getWithTs() (or something better)
>>>>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>>>> method returning KeyContext not be sufficient?
>>>>> - for backward compatibility, we will also need a new interface and
>>>>> cannot just extend the existing one
>>>>>
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 5/29/17 4:55 PM, Jeyhun Karimov wrote:
>>>>>> Dear community,
>>>>>>
>>>>>> I want to share KIP-165 [1] based on issue KAFKA-4304 [2].
>>>>>> I would like to get your comments.
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>> 165%3A+Extend+Interactive+Queries+for+return+latest+
>>>>> update+timestamp+per+key
>>>>>> [2] https://issues.apache.org/jira/browse/KAFKA-4304
>>>>>>
>>>>>> Cheers,
>>>>>> Jeyhun
>>>>>>
>>>>>
>>>
>>> --
>>> <http://www.openbet.com/> Michal Borowiecki
>>> Senior Software Engineer L4
>>> T: +44 208 742 1600 <tel:+44%2020%208742%201600>
>>>
>>>
>>> +44 203 249 8448 <tel:+44%2020%203249%208448>
>>>
>>>
>>>
>>> E: michal.borowiecki@openbet.com
>>> <ma...@openbet.com>
>>> W: www.openbet.com <http://www.openbet.com/>
>>>
>>>
>>> OpenBet Ltd
>>>
>>> Chiswick Park Building 9
>>>
>>> 566 Chiswick High Rd
>>>
>>> London
>>>
>>> W4 5XT
>>>
>>> UK
>>>
>>>
>>> <https://www.openbet.com/email_promo>
>>>
>>> This message is confidential and intended only for the addressee. If
>>> you have received this message in error, please immediately notify
>>> the postmaster@openbet.com <ma...@openbet.com> and
>>> delete it from your system as well as any copies. The content of
>>> e-mails as well as traffic data may be monitored by OpenBet for
>>> employment and security purposes. To protect the environment please
>>> do not print this e-mail unless necessary. OpenBet Ltd. Registered
>>> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
>>> 5XT, United Kingdom. A company registered in England and Wales.
>>> Registered no. 3134634. VAT no. GB927523612
>>>
>>> --
>>> -Cheers
>>>
>>> Jeyhun
>>
>
Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest
update timestamp per key
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Jeyhun,
is there any update regarding this KIP?
-Matthias
On 10/13/17 10:39 AM, Matthias J. Sax wrote:
> I wanted to follow up with this KIP.
>
> One important thing that is missing in this KIP, is the question where
> the TS is coming from? ATM, timestamps are not stored in the state
> stores -- it's a plain <key,value> format.
>
> Thus, if we want to enable this feature, we need to change the data
> format of the stores. This also required an upgrade path for
> applications to migrate from the old format to the new format.
>
>
> -Matthias
>
> On 6/27/17 2:16 PM, Jeyhun Karimov wrote:
>> Hi Michal,
>>
>>
>> Thanks a lot for your comment. I fixed the document.
>>
>> Cheers,
>> Jeyhun
>>
>> On Sat, Jun 24, 2017 at 6:49 PM Michal Borowiecki
>> <michal.borowiecki@openbet.com <ma...@openbet.com>>
>> wrote:
>>
>> Hi Jeyhun,
>>
>> Could the proposed KeyContext.keyTs() be made more descriptive?
>>
>> e.g. lastUpdated() or similar? So that users don't have to read the
>> docs to know it isn't the creation timestamp for instance.
>>
>> Cheers,
>> Michał
>>
>>
>> On 04/06/17 01:24, Jeyhun Karimov wrote:
>>> Hi Matthias,
>>>
>>> Thanks for comments.
>>>
>>> - why do you only consider get() and not range() and all() ?
>>>
>>>
>>> The corresponding jira concentrates on single key lookups. Moreover, I
>>> could not find a use-case to include range queries to return records with
>>> timestamp. However, theoritically we can include range() and all() as well.
>>>
>>> - we cannot have a second get() (this would be ambiguous) but need
>>>> another name like getWithTs() (or something better)
>>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>>> method returning KeyContext not be sufficient?
>>>
>>> Thanks for correction, this is my bad.
>>>
>>> - for backward compatibility, we will also need a new interface and
>>>> cannot just extend the existing one
>>>
>>> I will correct the KIP accordingly.
>>>
>>> Thanks,
>>> Jeyhun
>>>
>>> On Fri, Jun 2, 2017 at 7:36 AM, Matthias J. Sax <ma...@confluent.io> <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks for the KIP Jeyhun.
>>>>
>>>> Some comments:
>>>> - why do you only consider get() and not range() and all() ?
>>>> - we cannot have a second get() (this would be ambiguous) but need
>>>> another name like getWithTs() (or something better)
>>>> - what use case do you have in mind for getKeyTs() ? Would a single new
>>>> method returning KeyContext not be sufficient?
>>>> - for backward compatibility, we will also need a new interface and
>>>> cannot just extend the existing one
>>>>
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 5/29/17 4:55 PM, Jeyhun Karimov wrote:
>>>>> Dear community,
>>>>>
>>>>> I want to share KIP-165 [1] based on issue KAFKA-4304 [2].
>>>>> I would like to get your comments.
>>>>>
>>>>> [1]
>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>> 165%3A+Extend+Interactive+Queries+for+return+latest+
>>>> update+timestamp+per+key
>>>>> [2] https://issues.apache.org/jira/browse/KAFKA-4304
>>>>>
>>>>> Cheers,
>>>>> Jeyhun
>>>>>
>>>>
>>
>> --
>> <http://www.openbet.com/> Michal Borowiecki
>> Senior Software Engineer L4
>> T: +44 208 742 1600 <tel:+44%2020%208742%201600>
>>
>>
>> +44 203 249 8448 <tel:+44%2020%203249%208448>
>>
>>
>>
>> E: michal.borowiecki@openbet.com
>> <ma...@openbet.com>
>> W: www.openbet.com <http://www.openbet.com/>
>>
>>
>> OpenBet Ltd
>>
>> Chiswick Park Building 9
>>
>> 566 Chiswick High Rd
>>
>> London
>>
>> W4 5XT
>>
>> UK
>>
>>
>> <https://www.openbet.com/email_promo>
>>
>> This message is confidential and intended only for the addressee. If
>> you have received this message in error, please immediately notify
>> the postmaster@openbet.com <ma...@openbet.com> and
>> delete it from your system as well as any copies. The content of
>> e-mails as well as traffic data may be monitored by OpenBet for
>> employment and security purposes. To protect the environment please
>> do not print this e-mail unless necessary. OpenBet Ltd. Registered
>> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4
>> 5XT, United Kingdom. A company registered in England and Wales.
>> Registered no. 3134634. VAT no. GB927523612
>>
>> --
>> -Cheers
>>
>> Jeyhun
>