You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gary Ogden <go...@gmail.com> on 2016/10/03 13:51:31 UTC
kafka streams with dynamic content and filtering
I have a use case, and I'm wondering if it's possible to do this with Kafka.
Let's say we will have customers that will be uploading JSON to our system,
but that JSON layout will be different between each customer. They are able
to define the schema of the JSON being uploaded.
They will then be able to define the fields in that JSON they want to
gather metrics on (sum, counts etc).
Is there a way with Kafka streaming to dynamically read the configuration
for that customer and process the json and do counts and sums for the
fields they've defined.
It's possible at any time they may want to modify the configuration for
their json as well. Stop counting one field, start counting another.
They will also want to do some inferences as well. IE, if this particular
JSON is uploaded with a field in it, then check to see if another json was
uploaded within 8 hours.
Is it possible for Kafka streaming to be this dynamic?
Re: kafka streams with dynamic content and filtering
Posted by Gary Ogden <go...@gmail.com>.
Sorry. I responded to the wrong message....
On 5 October 2016 at 09:40, Gary Ogden <go...@gmail.com> wrote:
> Thanks Guozhang.
>
> So there's no way we could also use InternalTopicManager to specify the
> number of partitions and RF?
>
> https://github.com/apache/kafka/blob/0.10.1/streams/src/
> main/java/org/apache/kafka/streams/processor/internals/
> InternalTopicManager.java
>
>
>
> On 4 October 2016 at 19:25, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hello Gary,
>>
>> What you described should be workable with the lower-level Processor
>> interface of Kafka Streams, i.e. dynamic aggregations based on the input
>> data indicating changes to the JSON schemas. For detailed examples of how
>> the Processor API works please read the corresponding sections on the web
>> docs:
>>
>> http://docs.confluent.io/3.0.1/streams/developer-guide.html#processor-api
>>
>>
>> Guozhang
>>
>> On Mon, Oct 3, 2016 at 6:51 AM, Gary Ogden <go...@gmail.com> wrote:
>>
>> > I have a use case, and I'm wondering if it's possible to do this with
>> > Kafka.
>> >
>> > Let's say we will have customers that will be uploading JSON to our
>> system,
>> > but that JSON layout will be different between each customer. They are
>> able
>> > to define the schema of the JSON being uploaded.
>> >
>> > They will then be able to define the fields in that JSON they want to
>> > gather metrics on (sum, counts etc).
>> >
>> > Is there a way with Kafka streaming to dynamically read the
>> configuration
>> > for that customer and process the json and do counts and sums for the
>> > fields they've defined.
>> >
>> > It's possible at any time they may want to modify the configuration for
>> > their json as well. Stop counting one field, start counting another.
>> >
>> > They will also want to do some inferences as well. IE, if this
>> particular
>> > JSON is uploaded with a field in it, then check to see if another json
>> was
>> > uploaded within 8 hours.
>> >
>> > Is it possible for Kafka streaming to be this dynamic?
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
Re: kafka streams with dynamic content and filtering
Posted by Gary Ogden <go...@gmail.com>.
Thanks Guozhang.
So there's no way we could also use InternalTopicManager to specify the
number of partitions and RF?
https://github.com/apache/kafka/blob/0.10.1/streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java
On 4 October 2016 at 19:25, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Gary,
>
> What you described should be workable with the lower-level Processor
> interface of Kafka Streams, i.e. dynamic aggregations based on the input
> data indicating changes to the JSON schemas. For detailed examples of how
> the Processor API works please read the corresponding sections on the web
> docs:
>
> http://docs.confluent.io/3.0.1/streams/developer-guide.html#processor-api
>
>
> Guozhang
>
> On Mon, Oct 3, 2016 at 6:51 AM, Gary Ogden <go...@gmail.com> wrote:
>
> > I have a use case, and I'm wondering if it's possible to do this with
> > Kafka.
> >
> > Let's say we will have customers that will be uploading JSON to our
> system,
> > but that JSON layout will be different between each customer. They are
> able
> > to define the schema of the JSON being uploaded.
> >
> > They will then be able to define the fields in that JSON they want to
> > gather metrics on (sum, counts etc).
> >
> > Is there a way with Kafka streaming to dynamically read the configuration
> > for that customer and process the json and do counts and sums for the
> > fields they've defined.
> >
> > It's possible at any time they may want to modify the configuration for
> > their json as well. Stop counting one field, start counting another.
> >
> > They will also want to do some inferences as well. IE, if this particular
> > JSON is uploaded with a field in it, then check to see if another json
> was
> > uploaded within 8 hours.
> >
> > Is it possible for Kafka streaming to be this dynamic?
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka streams with dynamic content and filtering
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Gary,
What you described should be workable with the lower-level Processor
interface of Kafka Streams, i.e. dynamic aggregations based on the input
data indicating changes to the JSON schemas. For detailed examples of how
the Processor API works please read the corresponding sections on the web
docs:
http://docs.confluent.io/3.0.1/streams/developer-guide.html#processor-api
Guozhang
On Mon, Oct 3, 2016 at 6:51 AM, Gary Ogden <go...@gmail.com> wrote:
> I have a use case, and I'm wondering if it's possible to do this with
> Kafka.
>
> Let's say we will have customers that will be uploading JSON to our system,
> but that JSON layout will be different between each customer. They are able
> to define the schema of the JSON being uploaded.
>
> They will then be able to define the fields in that JSON they want to
> gather metrics on (sum, counts etc).
>
> Is there a way with Kafka streaming to dynamically read the configuration
> for that customer and process the json and do counts and sums for the
> fields they've defined.
>
> It's possible at any time they may want to modify the configuration for
> their json as well. Stop counting one field, start counting another.
>
> They will also want to do some inferences as well. IE, if this particular
> JSON is uploaded with a field in it, then check to see if another json was
> uploaded within 8 hours.
>
> Is it possible for Kafka streaming to be this dynamic?
>
--
-- Guozhang