You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Shrijit Pillai <pi...@gmail.com> on 2019/01/03 19:50:44 UTC

Child jobs not kicking off using SparkRunner in cluster mode

Hello,

I'm trying to run the WordCount example using SparkRunner.

In the client-mode, the child jobs are kicked off and the output is also produced. However in the cluster mode, the child jobs are not starting and no output is produced. 
I'm using Beam 2.9.0 and Spark 2.3.0

Here are the commands that I ran:-

Cluster-mode (No output produced)
spark2-submit --conf spark.driver.userClassPathFirst=true --class org.wordcount.WordCountSpark --master yarn --queue test --deploy-mode cluster  wordcount-0.1.0-SNAPSHOT-shaded.jar --runner=SparkRunner

Client-mode (Output produced)
spark2-submit --conf spark.driver.userClassPathFirst=true --class org.wordcount.WordCountSpark --master yarn --queue test --deploy-mode client  wordcount-0.1.0-SNAPSHOT-shaded.jar --runner=SparkRunner

Here's the code
https://gist.github.com/shrijitpillai/852a3c737c922af9e9ba4cd4c53b8530

Here's the mvn dependency tree:-
https://gist.github.com/shrijitpillai/f06797238cf6db9bf350e6a835fb34d6

Cluster mode yarn log
https://gist.github.com/shrijitpillai/2f1963ece26c152a2d68bc6ae8913fea

Can somebody help in identifying what is that I'm missing here?

Thanks,
Shrijit Pillai