You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kyle Winkelman (Jira)" <ji...@apache.org> on 2019/08/27 16:53:00 UTC

[jira] [Commented] (BEAM-2466) Add Kafka Streams runner

    [ https://issues.apache.org/jira/browse/BEAM-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916879#comment-16916879 ] 

Kyle Winkelman commented on BEAM-2466:
--------------------------------------

Hey everyone,

I have been working on my own POC: [https://github.com/kyle-winkelman/beam/tree/kafka-streams]

I'm getting closer. Simple stuff is working. Definitely have a lot of work ahead.

Some info:
 * Went for Impulse + SplittableDoFn for reading all sources (plan to do a custom implementation for KafkaIO)
 * GroupByKey uses KStream.through to get all KVs with the same key to the same nodes then uses KeyedWorkItems and SystemReduceFn to groupAlsoByWindow
 * ParDo uses a Transformer to transform input records and a Punctuator to evaluate timers.
 * SideInputs/PCollectionViews will be created by writing the data to a topic with the key being the StateNamespace.stringKey() that topic will then be read into a GlobalKTable and Materialized into a KeyValueStore. When a ParDo accesses the SideInput, we will give the Transformer access to the KeyValueStore that has the data. A call to get will look up the value for the corresponding window in the SideInput.
 * StateInternals and TimerInternals are materialized in KeyValueStores.

I plan to keep working away at it but if anyone is interested in collaborating the help will be much appreciated.

> Add Kafka Streams runner
> ------------------------
>
>                 Key: BEAM-2466
>                 URL: https://issues.apache.org/jira/browse/BEAM-2466
>             Project: Beam
>          Issue Type: Wish
>          Components: runner-ideas
>            Reporter: Lorand Peter Kasler
>            Assignee: Kai Jiang
>            Priority: Minor
>
> Kafka Streams (https://kafka.apache.org/documentation/streams)  has more and more features that could make it a viable candidate for a streaming runner. It uses DataFlow-like model



--
This message was sent by Atlassian Jira
(v8.3.2#803003)