You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Dongwon Kim <ea...@gmail.com> on 2020/11/16 23:32:32 UTC

How to run a per-job cluster for a Beam pipeline w/ FlinkRunner on YARN

Hi,

I'm trying to run a per-job cluster for a Beam pipeline w/ FlinkRunner on
YARN as follows:

> flink run -m yarn-cluster -d \

    my-beam-pipeline.jar \
>     --runner=FlinkRunner \
>     --flinkMaster=[auto] \
>     --parallelism=8


Instead of creating a per-job cluster as wished, the above command seems to
create a session cluster and then submit a job onto the cluster.

I doubt it because
(1) In the attached file, there's "Submit New Job" which is not shown in
other per-job applications that are written in Flink APIs and submitted to
YARN similar to the above command.
[image: beam on yarn.png]

(2) When the job is finished, the YARN application is still in its RUNNING
state without being terminated. I had to kill the YARN application manually.

FYI, I'm using
- Beam v2.24.0 (Flink 1.10)
- Hadoop v3.1.1

Thanks in advance,

Best,

Dongwon