[jira] [Created] (BEAM-7186) LTS backport: Issue with setting the parallelism at client level using Flink runner

["Maximilian Michels (JIRA)" <ji...@apache.org>]

Maximilian Michels created BEAM-7186:
----------------------------------------

             Summary: LTS backport: Issue with setting the parallelism at client level using Flink runner
                 Key: BEAM-7186
                 URL: https://issues.apache.org/jira/browse/BEAM-7186
             Project: Beam
          Issue Type: Bug
          Components: runner-flink
    Affects Versions: 2.0.0
         Environment: I am using Flink 1.2.1 running on Docker, with Task Managers distributed across different VMs as part of a Docker Swarm.
            Reporter: Maximilian Michels
            Assignee: Maximilian Michels
             Fix For: 2.8.0


When uploading an Apache Beam application using the Flink Web UI, the parallelism set at job submission doesn't get picked up. The same happens when submitting a job using the Flink CLI.

In both cases, the parallelism ends up defaulting to 1.

When I set the parallelism programmatically within the Apache Beam code, it works: {{flinkPipelineOptions.setParallelism(4);}}

I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values passed to Flink, then defaults to 1 if it doesn't find anything.

Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism dynamically, so the programmatic approach won't really work for me, nor will setting the Flink configuration at system level.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)