[jira] [Created] (BEAM-13682) Pipeline fails to start when .withDynamicRead() is activated

["Adrià Arcarons (Jira)" <ji...@apache.org>]

Adrià Arcarons created BEAM-13682:
-------------------------------------

             Summary: Pipeline fails to start when .withDynamicRead() is activated
                 Key: BEAM-13682
                 URL: https://issues.apache.org/jira/browse/BEAM-13682
             Project: Beam
          Issue Type: Bug
          Components: io-java-gcp
    Affects Versions: 2.35.0, 2.34.0
            Reporter: Adrià Arcarons
         Attachments: Main.java

We have a {+}Java{+}, {+}streaming{+}, pipeline that consumes messages from Kafka and ingests them into BigQuery. We run this pipeline in Dataflow.

Recently, we enabled the `.withDynamicRead()` option in our KafkaIO source to be able to auto-discover new partitions.

As a result, the pipeline won't load in Dataflow with the following (not very informative) error:
{noformat}
"Workflow failed. Causes: Internal Issue (fc4aaa0289f1f666): 62242584:26". 
{noformat}
This error happens before the actual job graph can be generated.

After contacting Google Support, they concluded that:
{noformat}
it is a bug in KafkaIO as ‘withDynamicReads is supposed to produce unique side input but it is not what is doing in your case
{noformat}
They also provided us with an internal Dataflow log message that can help to narrow the problem:
{noformat}
"invalid_argument: Duplicate side input name.  Side input names must be unique."
{noformat}
Some important pieces of information:
 * The error only happens if the destination sink is BigQueryIO. We have tested it by replacing BigQueryIO with TextIO and it works OK.
 * The error doesn't happen if pipeline is run by DirectRunner; only DataflowRunner.
 * We have reproduced this error in Beam SDK versions 2.34 and 2.35.

 

I've written a small pipeline that reproduces the error. Please find it attached in the ticket.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)