You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vasiliki Papavasileiou <vp...@confluent.io.INVALID> on 2021/11/29 12:20:10 UTC
[DISCUSS] KIP-805: Add range and scan query support in IQ v2
Hello everyone,
I would like to start the discussion for KIP-805: Add range and scan query
support in IQ v2
The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
Any suggestions are more than welcome.
Many thanks,
Vicky
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by Vasiliki Papavasileiou <vp...@confluent.io.INVALID>.
Hey Guozhang,
Thank you for looking into the KIP.
Windowed stores are addressed in another KIP.
FYI, I made a change to the KIP and removed the `RawRangeQuery`. After some
more thought, it doesn't provide us with many benefits (we save on one
cast) which doesn't justify the cost of adding an extra query to the public
interface.
On Thu, Dec 9, 2021 at 9:50 PM Guozhang Wang <wa...@gmail.com> wrote:
> Hi Vicky,
>
> Thanks for the KIP. Just for a bit more clarification, could you elaborate
> an example for windowed stores, beyond a key-value store (I think the
> `myStore` is for kv-store right?). Otherwise LGTM.
>
>
> Guozhang
>
> On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax <mj...@apache.org> wrote:
>
> > Thanks for the details!
> >
> > I also chatted with John about it, and he filed
> > https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some
> > feedback as follow up work.
> >
> > IMHO, the hard coded query translation is not ideal and should be
> > plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation
> > seems to be good enough.
> >
> >
> > -Matthias
> >
> > On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote:
> > > Hey Matthias,
> > >
> > > Thank you for looking into the KIP!
> > >
> > > We are adding raw versions of typed queries, like `RawRangeQuery`
> because
> > > it simplifies internal query handling since the bytes stores only
> support
> > > raw queries. A typed RangeQuery is handled by the `MeteredStore` which
> > > creates a new `RawRangeQuery` to pass down to the wrapped stores. When
> it
> > > gets the result back, it deserializes the data and creates a typed
> query
> > > result to return to the user. So, the store's key serde are used to
> > > translate typed `RangeQueries` into `RawRangeQueries` and it's value
> > serde
> > > are used to translate the result of the query on the way back. This
> > allows
> > > users to provide their own queries even if the MeteredStore has no
> > > knowledge of them.
> > >
> > > I hope this answers your question. Let me know if you have any other
> > > questions.
> > >
> > > Best,
> > > Vicky
> > >
> > >
> > > On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > >> Thanks for the KIP. Overall, make sense.
> > >>
> > >> One question: What is the purpose to `RawRangeQuery`? Seems not very
> > >> user friendly.
> > >>
> > >> -Matthias
> > >>
> > >>
> > >> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote:
> > >>> Thank you John! Yes, that was a typo from copying and I fixed it.
> > >>>
> > >>> Since there have been no more comments, I will start the vote.
> > >>>
> > >>> Best,
> > >>> Vicky
> > >>>
> > >>> On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org>
> > >> wrote:
> > >>>
> > >>>> Thanks for the KIP, Vicky!
> > >>>>
> > >>>> This KIP will help fill in the parity gap between IQ and
> > >>>> IQv2.
> > >>>>
> > >>>> One thing I noticed, which looks like just a typo is that
> > >>>> the value type of the proposed RangeQuery should probably be
> > >>>> KeyValueIterator<K, V>, right?
> > >>>>
> > >>>> Otherwise, it looks good to me!
> > >>>>
> > >>>> Thanks,
> > >>>> -John
> > >>>>
> > >>>> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
> > >>>> wrote:
> > >>>>> Hello everyone,
> > >>>>>
> > >>>>> I would like to start the discussion for KIP-805: Add range and
> scan
> > >>>> query
> > >>>>> support in IQ v2
> > >>>>>
> > >>>>> The KIP can be found here:
> > >>>>>
> > >>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
> > >>>>>
> > >>>>> Any suggestions are more than welcome.
> > >>>>>
> > >>>>> Many thanks,
> > >>>>> Vicky
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>
>
> --
> -- Guozhang
>
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Vicky,
Thanks for the KIP. Just for a bit more clarification, could you elaborate
an example for windowed stores, beyond a key-value store (I think the
`myStore` is for kv-store right?). Otherwise LGTM.
Guozhang
On Wed, Dec 8, 2021 at 4:18 PM Matthias J. Sax <mj...@apache.org> wrote:
> Thanks for the details!
>
> I also chatted with John about it, and he filed
> https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some
> feedback as follow up work.
>
> IMHO, the hard coded query translation is not ideal and should be
> plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation
> seems to be good enough.
>
>
> -Matthias
>
> On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote:
> > Hey Matthias,
> >
> > Thank you for looking into the KIP!
> >
> > We are adding raw versions of typed queries, like `RawRangeQuery` because
> > it simplifies internal query handling since the bytes stores only support
> > raw queries. A typed RangeQuery is handled by the `MeteredStore` which
> > creates a new `RawRangeQuery` to pass down to the wrapped stores. When it
> > gets the result back, it deserializes the data and creates a typed query
> > result to return to the user. So, the store's key serde are used to
> > translate typed `RangeQueries` into `RawRangeQueries` and it's value
> serde
> > are used to translate the result of the query on the way back. This
> allows
> > users to provide their own queries even if the MeteredStore has no
> > knowledge of them.
> >
> > I hope this answers your question. Let me know if you have any other
> > questions.
> >
> > Best,
> > Vicky
> >
> >
> > On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> Thanks for the KIP. Overall, make sense.
> >>
> >> One question: What is the purpose to `RawRangeQuery`? Seems not very
> >> user friendly.
> >>
> >> -Matthias
> >>
> >>
> >> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote:
> >>> Thank you John! Yes, that was a typo from copying and I fixed it.
> >>>
> >>> Since there have been no more comments, I will start the vote.
> >>>
> >>> Best,
> >>> Vicky
> >>>
> >>> On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org>
> >> wrote:
> >>>
> >>>> Thanks for the KIP, Vicky!
> >>>>
> >>>> This KIP will help fill in the parity gap between IQ and
> >>>> IQv2.
> >>>>
> >>>> One thing I noticed, which looks like just a typo is that
> >>>> the value type of the proposed RangeQuery should probably be
> >>>> KeyValueIterator<K, V>, right?
> >>>>
> >>>> Otherwise, it looks good to me!
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
> >>>> wrote:
> >>>>> Hello everyone,
> >>>>>
> >>>>> I would like to start the discussion for KIP-805: Add range and scan
> >>>> query
> >>>>> support in IQ v2
> >>>>>
> >>>>> The KIP can be found here:
> >>>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
> >>>>>
> >>>>> Any suggestions are more than welcome.
> >>>>>
> >>>>> Many thanks,
> >>>>> Vicky
> >>>>
> >>>>
> >>>
> >>
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by "Matthias J. Sax" <mj...@apache.org>.
Thanks for the details!
I also chatted with John about it, and he filed
https://issues.apache.org/jira/browse/KAFKA-13526 to incorporate some
feedback as follow up work.
IMHO, the hard coded query translation is not ideal and should be
plugable. But for a v1 of IQv2 (pun intended) the hardcoded translation
seems to be good enough.
-Matthias
On 12/8/21 9:37 AM, Vasiliki Papavasileiou wrote:
> Hey Matthias,
>
> Thank you for looking into the KIP!
>
> We are adding raw versions of typed queries, like `RawRangeQuery` because
> it simplifies internal query handling since the bytes stores only support
> raw queries. A typed RangeQuery is handled by the `MeteredStore` which
> creates a new `RawRangeQuery` to pass down to the wrapped stores. When it
> gets the result back, it deserializes the data and creates a typed query
> result to return to the user. So, the store's key serde are used to
> translate typed `RangeQueries` into `RawRangeQueries` and it's value serde
> are used to translate the result of the query on the way back. This allows
> users to provide their own queries even if the MeteredStore has no
> knowledge of them.
>
> I hope this answers your question. Let me know if you have any other
> questions.
>
> Best,
> Vicky
>
>
> On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax <mj...@apache.org> wrote:
>
>> Thanks for the KIP. Overall, make sense.
>>
>> One question: What is the purpose to `RawRangeQuery`? Seems not very
>> user friendly.
>>
>> -Matthias
>>
>>
>> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote:
>>> Thank you John! Yes, that was a typo from copying and I fixed it.
>>>
>>> Since there have been no more comments, I will start the vote.
>>>
>>> Best,
>>> Vicky
>>>
>>> On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org>
>> wrote:
>>>
>>>> Thanks for the KIP, Vicky!
>>>>
>>>> This KIP will help fill in the parity gap between IQ and
>>>> IQv2.
>>>>
>>>> One thing I noticed, which looks like just a typo is that
>>>> the value type of the proposed RangeQuery should probably be
>>>> KeyValueIterator<K, V>, right?
>>>>
>>>> Otherwise, it looks good to me!
>>>>
>>>> Thanks,
>>>> -John
>>>>
>>>> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
>>>> wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I would like to start the discussion for KIP-805: Add range and scan
>>>> query
>>>>> support in IQ v2
>>>>>
>>>>> The KIP can be found here:
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
>>>>>
>>>>> Any suggestions are more than welcome.
>>>>>
>>>>> Many thanks,
>>>>> Vicky
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by Vasiliki Papavasileiou <vp...@confluent.io.INVALID>.
Hey Matthias,
Thank you for looking into the KIP!
We are adding raw versions of typed queries, like `RawRangeQuery` because
it simplifies internal query handling since the bytes stores only support
raw queries. A typed RangeQuery is handled by the `MeteredStore` which
creates a new `RawRangeQuery` to pass down to the wrapped stores. When it
gets the result back, it deserializes the data and creates a typed query
result to return to the user. So, the store's key serde are used to
translate typed `RangeQueries` into `RawRangeQueries` and it's value serde
are used to translate the result of the query on the way back. This allows
users to provide their own queries even if the MeteredStore has no
knowledge of them.
I hope this answers your question. Let me know if you have any other
questions.
Best,
Vicky
On Tue, Dec 7, 2021 at 12:46 AM Matthias J. Sax <mj...@apache.org> wrote:
> Thanks for the KIP. Overall, make sense.
>
> One question: What is the purpose to `RawRangeQuery`? Seems not very
> user friendly.
>
> -Matthias
>
>
> On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote:
> > Thank you John! Yes, that was a typo from copying and I fixed it.
> >
> > Since there have been no more comments, I will start the vote.
> >
> > Best,
> > Vicky
> >
> > On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org>
> wrote:
> >
> >> Thanks for the KIP, Vicky!
> >>
> >> This KIP will help fill in the parity gap between IQ and
> >> IQv2.
> >>
> >> One thing I noticed, which looks like just a typo is that
> >> the value type of the proposed RangeQuery should probably be
> >> KeyValueIterator<K, V>, right?
> >>
> >> Otherwise, it looks good to me!
> >>
> >> Thanks,
> >> -John
> >>
> >> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
> >> wrote:
> >>> Hello everyone,
> >>>
> >>> I would like to start the discussion for KIP-805: Add range and scan
> >> query
> >>> support in IQ v2
> >>>
> >>> The KIP can be found here:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
> >>>
> >>> Any suggestions are more than welcome.
> >>>
> >>> Many thanks,
> >>> Vicky
> >>
> >>
> >
>
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by "Matthias J. Sax" <mj...@apache.org>.
Thanks for the KIP. Overall, make sense.
One question: What is the purpose to `RawRangeQuery`? Seems not very
user friendly.
-Matthias
On 11/30/21 12:48 PM, Vasiliki Papavasileiou wrote:
> Thank you John! Yes, that was a typo from copying and I fixed it.
>
> Since there have been no more comments, I will start the vote.
>
> Best,
> Vicky
>
> On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org> wrote:
>
>> Thanks for the KIP, Vicky!
>>
>> This KIP will help fill in the parity gap between IQ and
>> IQv2.
>>
>> One thing I noticed, which looks like just a typo is that
>> the value type of the proposed RangeQuery should probably be
>> KeyValueIterator<K, V>, right?
>>
>> Otherwise, it looks good to me!
>>
>> Thanks,
>> -John
>>
>> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
>> wrote:
>>> Hello everyone,
>>>
>>> I would like to start the discussion for KIP-805: Add range and scan
>> query
>>> support in IQ v2
>>>
>>> The KIP can be found here:
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
>>>
>>> Any suggestions are more than welcome.
>>>
>>> Many thanks,
>>> Vicky
>>
>>
>
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by Vasiliki Papavasileiou <vp...@confluent.io.INVALID>.
Thank you John! Yes, that was a typo from copying and I fixed it.
Since there have been no more comments, I will start the vote.
Best,
Vicky
On Tue, Nov 30, 2021 at 5:22 AM John Roesler <vv...@apache.org> wrote:
> Thanks for the KIP, Vicky!
>
> This KIP will help fill in the parity gap between IQ and
> IQv2.
>
> One thing I noticed, which looks like just a typo is that
> the value type of the proposed RangeQuery should probably be
> KeyValueIterator<K, V>, right?
>
> Otherwise, it looks good to me!
>
> Thanks,
> -John
>
> On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
> wrote:
> > Hello everyone,
> >
> > I would like to start the discussion for KIP-805: Add range and scan
> query
> > support in IQ v2
> >
> > The KIP can be found here:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
> >
> > Any suggestions are more than welcome.
> >
> > Many thanks,
> > Vicky
>
>
Re: [DISCUSS] KIP-805: Add range and scan query support in IQ v2
Posted by John Roesler <vv...@apache.org>.
Thanks for the KIP, Vicky!
This KIP will help fill in the parity gap between IQ and
IQv2.
One thing I noticed, which looks like just a typo is that
the value type of the proposed RangeQuery should probably be
KeyValueIterator<K, V>, right?
Otherwise, it looks good to me!
Thanks,
-John
On Mon, 2021-11-29 at 12:20 +0000, Vasiliki Papavasileiou
wrote:
> Hello everyone,
>
> I would like to start the discussion for KIP-805: Add range and scan query
> support in IQ v2
>
> The KIP can be found here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-805%3A+Add+range+and+scan+query+support+in+IQ+v2
>
> Any suggestions are more than welcome.
>
> Many thanks,
> Vicky