You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Maximilian Michels (JIRA)" <ji...@apache.org> on 2019/02/27 11:57:00 UTC

[jira] [Created] (BEAM-6751) KafkaIO blocks FlinkRunner in EOS mode

Maximilian Michels created BEAM-6751:
----------------------------------------

             Summary: KafkaIO blocks FlinkRunner in EOS mode
                 Key: BEAM-6751
                 URL: https://issues.apache.org/jira/browse/BEAM-6751
             Project: Beam
          Issue Type: Bug
          Components: io-java-kafka, runner-flink
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels
             Fix For: 2.12.0


KafkaIO has a validation check which whitelists certain runners capable of provide exactly-once semantics:
{noformat}
        if ("org.apache.beam.runners.direct.DirectRunner".equals(runner)
            || runner.startsWith("org.apache.beam.runners.dataflow.")
            || runner.startsWith("org.apache.beam.runners.spark.") {
...
{noformat}

The FlinkRunner supports exactly-once checkpointing but is blocked from using Kafka's exactly once mode. 

I wonder if such a list is easily maintainable? I think we should replace the list with a warning instead. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)