You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Maximilian Michels (JIRA)" <ji...@apache.org> on 2019/02/27 11:57:00 UTC
[jira] [Created] (BEAM-6751) KafkaIO blocks FlinkRunner in EOS mode
Maximilian Michels created BEAM-6751:
----------------------------------------
Summary: KafkaIO blocks FlinkRunner in EOS mode
Key: BEAM-6751
URL: https://issues.apache.org/jira/browse/BEAM-6751
Project: Beam
Issue Type: Bug
Components: io-java-kafka, runner-flink
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Fix For: 2.12.0
KafkaIO has a validation check which whitelists certain runners capable of provide exactly-once semantics:
{noformat}
if ("org.apache.beam.runners.direct.DirectRunner".equals(runner)
|| runner.startsWith("org.apache.beam.runners.dataflow.")
|| runner.startsWith("org.apache.beam.runners.spark.") {
...
{noformat}
The FlinkRunner supports exactly-once checkpointing but is blocked from using Kafka's exactly once mode.
I wonder if such a list is easily maintainable? I think we should replace the list with a warning instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)