You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/30 07:52:40 UTC

[GitHub] [beam] mosche opened a new issue, #23439: [Bug]: Spark "uber jar" job server fails on cluster with NoSuchFileException

mosche opened a new issue, #23439:
URL: https://github.com/apache/beam/issues/23439

   ### What happened?
   
   Attempting to use the Spark "uber jar" job server when not in local mode fails with a NoSuchFileException.
   ```
   python -m apache_beam.examples.wordcount \
   --output ./data_test/ \
   --runner=SparkRunner \
   --spark_submit_uber_jar \
   --spark_master_url=spark://spark-master:7077 \
   --environment_type=LOOPBACK
   ```
   
   This was reported in https://stackoverflow.com/a/66342031/4278032, I was able to reproduce the issue locally.
   
   ```
   ERROR:root:Exception from the cluster: java.nio.file.NoSuchFileException: /tmp/tmp029vgh_0.jar
   ```
   
   A work-around is to mount some shared storage for `/tmp` on all workers and the job itself. That suggests the jar isn't copied to all places where required.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: runner-spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Spark "uber jar" job server fails on cluster with NoSuchFileException [beam]

Posted by "jvjrood (via GitHub)" <gi...@apache.org>.
jvjrood commented on issue #23439:
URL: https://github.com/apache/beam/issues/23439#issuecomment-2652860661

   Issue still persists. Any updates?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] meowcakes commented on issue #23439: [Bug]: Spark "uber jar" job server fails on cluster with NoSuchFileException

Posted by "meowcakes (via GitHub)" <gi...@apache.org>.
meowcakes commented on issue #23439:
URL: https://github.com/apache/beam/issues/23439#issuecomment-1718693206

   Any updates on this? It still seems like an issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org