You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Olivier Sannier (JIRA)" <ji...@apache.org> on 2018/02/02 14:28:00 UTC
[jira] [Updated] (SPARK-23320) RANDOM pseudo environment variable
has low resolution under Windows
[ https://issues.apache.org/jira/browse/SPARK-23320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivier Sannier updated SPARK-23320:
------------------------------------
Description:
Under Windows, spark-submit.bat calls spark-class2.cmd which then runs org.apache.spark.launcher.Main to retrieve its output and place it into the SPARK_CMD variable.
To do so, it uses a redirection to a temporary file whose name is created with this command:
{{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
Note how it uses the %RANDOM% variable to get a hopefully unique name for the file that will get created.
There are two issues with this however:
# if we have bad luck, we can get the same value RANDOM
# bad luck is quite easy to get if we submit numerous jobs at once, because the granularity of RANDOM is based on the current second, as indicated [here|https://blogs.msdn.microsoft.com/oldnewthing/20100617-00/?p=13673].
When two concurrent spark-submit calls use the same generated file, one of them fails with a "file is in use" error message.
I would thus suggest replacing the above line with these ones:
{{rem Because random is based on the current time, second based, it may collide quite easily.}}
{{rem To avoid this, we retry until a non existent file name is found and we create that file as soon as possible}}
{{set laucher_try_count=5}}
{{:retry_launcher}}
{{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
{{if not exist "%LAUNCHER_OUTPUT%" goto :launcher_success}}
{{set /A laucher_try_count -= 1}}
{{if %laucher_try_count% GTR 0 goto :retry_launcher}}
{{echo Could not generate a launcher output filename that does not already exists 1>&2}}
{{goto :eof}}
{{:launcher_success}}
{{echo rem > %LAUNCHER_OUTPUT%}}
This code has been tried here and effectively solves the issue.
was:
Under Windows, spark-submit.bat calls spark-class2.cmd which then runs org.apache.spark.launcher.Main to retrieve its output and place it into the SPARK_CMD variable.
To do so, it uses a redirection to a temporary file whose name is created with this command:
{{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
Note how it uses the %RANDOM% variable to get a hopefully unique name for the file that will get created.
There are two issues with this however:
# if we have bad luck, we can get the same value RANDOM
# bad luck is quite easy to get if we submit numerous jobs at once, because the granularity of RANDOM is based on the current second, as indicated [here|https://blogs.msdn.microsoft.com/oldnewthing/20100617-00/?p=13673].
When two concurrent spark-submit calls use the same generated file, one of them fails with a "file is in use" error message.
I would thus suggest replacing the above lines with these ones:
{{rem Because random is based on the current time, second based, it may collide quite easily.}}
{{rem To avoid this, we retry until a non existent file name is found and we create that file as soon as possible}}
{{set laucher_try_count=5}}
{{:retry_launcher}}
{{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
{{if not exist "%LAUNCHER_OUTPUT%" goto :launcher_success}}
{{set /A laucher_try_count -= 1}}
{{if %laucher_try_count% GTR 0 goto :retry_launcher}}
{{echo Could not generate a launcher output filename that does not already exists 1>&2}}
{{goto :eof}}
{{:launcher_success}}
{{echo rem > %LAUNCHER_OUTPUT%}}
This code has been tried here and effectively solves the issue.
> RANDOM pseudo environment variable has low resolution under Windows
> -------------------------------------------------------------------
>
> Key: SPARK-23320
> URL: https://issues.apache.org/jira/browse/SPARK-23320
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.1.1
> Environment: Windows 7, Windows 10
> Spark 2.1.1
> Reporter: Olivier Sannier
> Priority: Minor
>
> Under Windows, spark-submit.bat calls spark-class2.cmd which then runs org.apache.spark.launcher.Main to retrieve its output and place it into the SPARK_CMD variable.
> To do so, it uses a redirection to a temporary file whose name is created with this command:
> {{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
> Note how it uses the %RANDOM% variable to get a hopefully unique name for the file that will get created.
> There are two issues with this however:
> # if we have bad luck, we can get the same value RANDOM
> # bad luck is quite easy to get if we submit numerous jobs at once, because the granularity of RANDOM is based on the current second, as indicated [here|https://blogs.msdn.microsoft.com/oldnewthing/20100617-00/?p=13673].
> When two concurrent spark-submit calls use the same generated file, one of them fails with a "file is in use" error message.
> I would thus suggest replacing the above line with these ones:
> {{rem Because random is based on the current time, second based, it may collide quite easily.}}
> {{rem To avoid this, we retry until a non existent file name is found and we create that file as soon as possible}}
> {{set laucher_try_count=5}}
> {{:retry_launcher}}
> {{set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt}}
> {{if not exist "%LAUNCHER_OUTPUT%" goto :launcher_success}}
> {{set /A laucher_try_count -= 1}}
> {{if %laucher_try_count% GTR 0 goto :retry_launcher}}
> {{echo Could not generate a launcher output filename that does not already exists 1>&2}}
> {{goto :eof}}
> {{:launcher_success}}
> {{echo rem > %LAUNCHER_OUTPUT%}}
> This code has been tried here and effectively solves the issue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org