You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Dongwon Kim <ea...@gmail.com> on 2020/11/16 23:32:32 UTC
How to run a per-job cluster for a Beam pipeline w/ FlinkRunner on YARN
Hi,
I'm trying to run a per-job cluster for a Beam pipeline w/ FlinkRunner on
YARN as follows:
> flink run -m yarn-cluster -d \
my-beam-pipeline.jar \
> --runner=FlinkRunner \
> --flinkMaster=[auto] \
> --parallelism=8
Instead of creating a per-job cluster as wished, the above command seems to
create a session cluster and then submit a job onto the cluster.
I doubt it because
(1) In the attached file, there's "Submit New Job" which is not shown in
other per-job applications that are written in Flink APIs and submitted to
YARN similar to the above command.
[image: beam on yarn.png]
(2) When the job is finished, the YARN application is still in its RUNNING
state without being terminated. I had to kill the YARN application manually.
FYI, I'm using
- Beam v2.24.0 (Flink 1.10)
- Hadoop v3.1.1
Thanks in advance,
Best,
Dongwon