You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Flávio Stutz <fl...@gmail.com> on 2018/06/25 17:13:12 UTC
[DISCUSS] KIP-323: Schedulable KTable as Graph source
Hey, guys, I've just started a KIP discussion here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-323%3A+Schedulable+KTable+as+Graph+source
Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source
Posted by .
Just to let you guys know, this KIP was actually registered under the number 326:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-326%3A+Schedulable+KTable+as+Graph+source
On 2018/06/25 17:13:12, Flávio Stutz <fl...@gmail.com> wrote:
> Hey, guys, I've just started a KIP discussion here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-323%3A+Schedulable+KTable+as+Graph+source
>
Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source
Posted by .
Guozhang,
Please go to https://lists.apache.org/thread.html/e93499c918df3a3985c6da8e0f240e286f7ba321b6e532705f994a59@%3Cdev.kafka.apache.org%3E
so we can continue this thread. Someone else got this KIP number while I was registering it, so the actual number is KIP-326. I already copied your reply contents to the new thread.
Thanks.
-Flávio Stutz
On 2018/06/25 22:24:17, Guozhang Wang <wa...@gmail.com> wrote:
> Flávio, thanks for creating this KIP.
>
> I think this "single-aggregation" use case is common enough that we should
> consider how to efficiently supports it: for example, for KSQL that's built
> on top of Streams, we've seen lots of query statements whose return is
> expected a single row indicating the "total aggregate" etc. See
> https://github.com/confluentinc/ksql/issues/430 for details.
>
> I've not read through https://issues.apache.org/jira/browse/KAFKA-6953, but
> I'm wondering if we have discussed the option of supporting it in a
> "pre-aggregate" manner: that is we do partial aggregates on parallel tasks,
> and then sends the partial aggregated value via a single topic partition
> for the final aggregate, to reduce the traffic on that single partition and
> hence the final aggregate workload.
> Of course, for non-commutative aggregates we'd probably need to provide
> another API in addition to aggregate, like the `merge` function for
> session-based aggregates, to let users customize the operations of merging
> two partial aggregates into a single partial aggregate. What's its pros and
> cons compared with the current proposal?
>
>
> Guozhang
>
> On Mon, Jun 25, 2018 at 3:12 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > This would be useful feature.
> >
> > In the Public Interfaces section, the new method lacks a closing
> > parenthesis.
> >
> > In the Proposed Changes section, if the order of the 3 bullets can match
> > the order of the parameters of the new method, it would be easier to read.
> >
> > For Rejected Alternatives #2, can you add a sentence saying why it was
> > rejected ?
> >
> > Cheers
> >
> > On Mon, Jun 25, 2018 at 10:13 AM, Flávio Stutz <fl...@gmail.com>
> > wrote:
> >
> > > Hey, guys, I've just started a KIP discussion here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 323%3A+Schedulable+KTable+as+Graph+source
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source
Posted by Guozhang Wang <wa...@gmail.com>.
Flávio, thanks for creating this KIP.
I think this "single-aggregation" use case is common enough that we should
consider how to efficiently supports it: for example, for KSQL that's built
on top of Streams, we've seen lots of query statements whose return is
expected a single row indicating the "total aggregate" etc. See
https://github.com/confluentinc/ksql/issues/430 for details.
I've not read through https://issues.apache.org/jira/browse/KAFKA-6953, but
I'm wondering if we have discussed the option of supporting it in a
"pre-aggregate" manner: that is we do partial aggregates on parallel tasks,
and then sends the partial aggregated value via a single topic partition
for the final aggregate, to reduce the traffic on that single partition and
hence the final aggregate workload.
Of course, for non-commutative aggregates we'd probably need to provide
another API in addition to aggregate, like the `merge` function for
session-based aggregates, to let users customize the operations of merging
two partial aggregates into a single partial aggregate. What's its pros and
cons compared with the current proposal?
Guozhang
On Mon, Jun 25, 2018 at 3:12 PM, Ted Yu <yu...@gmail.com> wrote:
> This would be useful feature.
>
> In the Public Interfaces section, the new method lacks a closing
> parenthesis.
>
> In the Proposed Changes section, if the order of the 3 bullets can match
> the order of the parameters of the new method, it would be easier to read.
>
> For Rejected Alternatives #2, can you add a sentence saying why it was
> rejected ?
>
> Cheers
>
> On Mon, Jun 25, 2018 at 10:13 AM, Flávio Stutz <fl...@gmail.com>
> wrote:
>
> > Hey, guys, I've just started a KIP discussion here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 323%3A+Schedulable+KTable+as+Graph+source
> >
>
--
-- Guozhang
Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source
Posted by Ted Yu <yu...@gmail.com>.
This would be useful feature.
In the Public Interfaces section, the new method lacks a closing
parenthesis.
In the Proposed Changes section, if the order of the 3 bullets can match
the order of the parameters of the new method, it would be easier to read.
For Rejected Alternatives #2, can you add a sentence saying why it was
rejected ?
Cheers
On Mon, Jun 25, 2018 at 10:13 AM, Flávio Stutz <fl...@gmail.com>
wrote:
> Hey, guys, I've just started a KIP discussion here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 323%3A+Schedulable+KTable+as+Graph+source
>