You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Huiming Jiang <hm...@samsung.com> on 2024/01/04 08:06:23 UTC
Beam on spark yarn could only start one worker
Hello,
We are using beam, which is written by python. And we want to deploy to
spark on yarn cluster .
But we found that it could only start one spark worker. Also on this worker,
beam docker is started.
\- The beam pipeline option printed :
{key:"beam:option:spark_master:v1" value:{string_value:"local[4]"}}
{key:"beam:option:direct_runner_use_stacked_bundle:v1"
value:{bool_value:true}}
Our steps:
Firstly, we generate jar file with python file.
python3.7 -m beam_demo \
\--spark_version=3 \
\--runner=SparkRunner \
\--spark_job_server_jar=/home/hadoop/beam-runners-spark-3-job-
server-2.52.0.jar \
\--sdk_container_image=apache/beam_python3.7_sdk_boto3:2.45.0 \
\--sdk_harness_container_image=apache/beam_python3.7_sdk_boto3:2.45.0 \
\--worker_harness_container_image=apache/beam_python3.7_sdk_boto3:2.45.0 \
\--environment_type=DOCKER
--environment_config=apache/beam_python3.7_sdk_boto3:2.45.0 \
\--sdk_location=container \
\--num_workers=2 \
\--output_executable_path=jars/beam_demo.jar
Secondly, we submit it to spark:
spark-submit \--master yarn --deploy-mode cluster jars/beam_demo.jar
---
Our question is:
1\. We configured sparkRunner. Why the runner is
direct_runner_use_stacked_bundle?
2\. We submitted job to spark on yarn. Why the printed spark_master is
local[4]?
3\. Could python beam on Spark yarn cluster start multi workers?
Could you please help to check those questions? Maybe we have something
missing. Thank you very much for your help.
---
---
---