You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Chamikara Madhusanka Jayalath (Jira)" <ji...@apache.org> on 2021/03/16 06:44:00 UTC
[jira] [Updated] (BEAM-11991) Python Kafka source not emitting
messages for streaming pipelines with Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chamikara Madhusanka Jayalath updated BEAM-11991:
-------------------------------------------------
Summary: Python Kafka source not emitting messages for streaming pipelines with Flink Runner (was: Python Kafka source not emitting messages for streaming pipelines)
> Python Kafka source not emitting messages for streaming pipelines with Flink Runner
> -----------------------------------------------------------------------------------
>
> Key: BEAM-11991
> URL: https://issues.apache.org/jira/browse/BEAM-11991
> Project: Beam
> Issue Type: Bug
> Components: cross-language, runner-flink
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Boyuan Zhang
> Priority: P1
>
> To reproduce:
> * Startup a local Kafka cluster: [https://kafka.apache.org/quickstart]
> * Setup topics:
> bin/kafka-console-consumer.sh --topic mytopic1 --from-beginning --bootstrap-server localhost:9092
> bin/kafka-console-consumer.sh --topic mytopic2 --from-beginning --bootstrap-server localhost:9092
> * Setup a Beam virtualenv and run a pipeline that reads from Kafka. For example: [https://wtools.io/paste-code/b4je]
> > python ./pipeline.py --bootstrap_servers=localhost:9092 --in_topic=mytopic1 --out_topic=mytopic2 --runner=FlinkRunner --streaming
> * Publish messages as kv pairs.
> ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic1 --property "parse.key=true" --property "key.separator=:"
> >a:b
> >c:d
> >e:f
> * Messages do not get pushed to subsequent steps.
> * Following seems to be working fine.
> * X-lang Bounded read with Flink
> * X-lang Kafka sink and with Flink
> [~boyuanz] could you take a look to rule out any SDF/unbounded read related issues ?
> cc: [~mxm] and [~angoenka] for Flink issues.
>
> Beam user thread: [https://lists.apache.org/x/thread.html/r9c74a8a7efa4b14f2f1d5f77ce8f12128cf1071861c1627f00702415@%3Cuser.beam.apache.org%3E]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)