You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hugh Zabriskie (JIRA)" <ji...@apache.org> on 2018/01/09 23:09:00 UTC
[jira] [Updated] (SPARK-23015) spark-submit fails when submitting
several jobs in parallel
[ https://issues.apache.org/jira/browse/SPARK-23015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hugh Zabriskie updated SPARK-23015:
-----------------------------------
Description:
Spark Submit's launching library prints the command to execute the launcher (org.apache.spark.launcher.main) to a temporary text file, reads the result back into a variable, and then executes that command.
[bin/spark-class2.cmd, L67|https://github.com/apache/spark/blob/master/bin/spark-class2.cmd#L66]
That temporary text file is given a pseudo-random name by the %RANDOM% env variable generator, which generates a number between 0 and 32767.
This appears to be the cause of an error occurring when several spark-submit jobs are launched simultaneously. The following error is returned from stderr:
"The process cannot access the file because it is being used by another process.The system cannot find the file {USER}\AppData\Local\Temp\spark-class-launcher-output-{RANDOM's int}.txt.The process cannot access the file because it is being used by another process."
My hypothesis is that %RANDOM% is returning the same value for multiple jobs, causing the launcher library to attempt to write to the same file from multiple processes. Another mechanism is needed for reliably generating the names of the temporary files so that the concurrency issue is resolved.
was:
Spark Submit's launching library prints the command to execute the launcher (org.apache.spark.launcher.main) to a temporary text file, reads the result back into a variable, and then executes that command.
[bin/spark-class2.cmd, L67|https://github.com/apache/spark/blob/master/bin/spark-class2.cmd#L67]
That temporary text file is given a pseudo-random name by the %RANDOM% env variable generator, which generates a number between 0 and 32767.
This appears to be the cause of an error occurring when several spark-submit jobs are launched simultaneously. The following error is returned from stderr:
"The process cannot access the file because it is being used by another process.The system cannot find the file {USER}\AppData\Local\Temp\spark-class-launcher-output-{RANDOM's int}.txt.The process cannot access the file because it is being used by another process."
My hypothesis is that %RANDOM% is returning the same value for multiple jobs, causing the launcher library to attempt to write to the same file from multiple processes. Another mechanism is needed for reliably generating the names of the temporary files so that the concurrency issue is resolved.
> spark-submit fails when submitting several jobs in parallel
> -----------------------------------------------------------
>
> Key: SPARK-23015
> URL: https://issues.apache.org/jira/browse/SPARK-23015
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.4.0, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1
> Environment: Windows 10 (1709/16299.125)
> Spark 2.3.0
> Java 8, Update 151
> Reporter: Hugh Zabriskie
>
> Spark Submit's launching library prints the command to execute the launcher (org.apache.spark.launcher.main) to a temporary text file, reads the result back into a variable, and then executes that command.
> [bin/spark-class2.cmd, L67|https://github.com/apache/spark/blob/master/bin/spark-class2.cmd#L66]
> That temporary text file is given a pseudo-random name by the %RANDOM% env variable generator, which generates a number between 0 and 32767.
> This appears to be the cause of an error occurring when several spark-submit jobs are launched simultaneously. The following error is returned from stderr:
> "The process cannot access the file because it is being used by another process.The system cannot find the file {USER}\AppData\Local\Temp\spark-class-launcher-output-{RANDOM's int}.txt.The process cannot access the file because it is being used by another process."
> My hypothesis is that %RANDOM% is returning the same value for multiple jobs, causing the launcher library to attempt to write to the same file from multiple processes. Another mechanism is needed for reliably generating the names of the temporary files so that the concurrency issue is resolved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org