You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/08 23:31:15 UTC
[GitHub] [spark] philwalk opened a new pull request, #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
philwalk opened a new pull request, #38167:
URL: https://github.com/apache/spark/pull/38167
This fixes two problems that affect development in a Windows shell environment, such as `cygwin` or `msys2`.
Running `./build/sbt packageBin` from A Windows cygwin `bash` session fails.
```bash
$ ./build.sbt packageBin
...
[info] compiling 9 Java sources to C:\Users\philwalk\workspace\spark\common\sketch\target\scala-2.12\classes ...
/bin/bash: C:Usersphilwalkworkspacesparkcore/../build/spark-build-info: No such file or directory
[info] compiling 1 Scala source to C:\Users\philwalk\workspace\spark\tools\target\scala-2.12\classes ...
[info] compiling 5 Scala sources to C:\Users\philwalk\workspace\spark\mllib-local\target\scala-2.12\classes ...
[info] Compiling 5 protobuf files to C:\Users\philwalk\workspace\spark\connector\connect\target\scala-2.12\src_managed\main
[error] stack trace is suppressed; run last core / Compile / managedResources for the full output
[error] (core / Compile / managedResources) Nonzero exit value: 127
[error] Total time: 42 s, completed Oct 8, 2022, 4:49:12 PM
sbt:spark-parent>
sbt:spark-parent> last core /Compile /managedResources
last core /Compile /managedResources
[error] java.lang.RuntimeException: Nonzero exit value: 127
[error] at scala.sys.package$.error(package.scala:30)
[error] at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
[error] at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
[error] at Core$.$anonfun$settings$4(SparkBuild.scala:604)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] at sbt.Execute.work(Execute.scala:291)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] at java.base/java.lang.Thread.run(Thread.java:834)
[error] (core / Compile / managedResources) Nonzero exit value: 127
```
This occurs if `WSL` is installed, because `project\SparkBuild.scala` creates a `bash` process, but `WSL bash` is called, even though `cygwin bash` appears earlier in the `PATH`, and because arguments to bash contain backslashes. The fix is to insure that the correct `bash` is called, and that arguments passed to `bash` are passed with backslashes rather than slashes.
The other problem fixed by the PR is to address problems preventing the `bash` scripts (`spark-shell`, `spark-submit`, etc.) from being used in Windows `SHELL` environments. The problem is that the bash version of `spark-class` fails in a Windows shell environment, the result of `launcher/src/main/java/org/apache/spark/launcher/Main.java` not following the convention expected by `spark-class`, and also appending CR to line endings.
There are two parts to this fix:
1. modify `Main.java` to treat a `SHELL` session on Windows as a `bash` session
2. remove the appended CR character when parsing the output produced by `Main.java`
### Does this PR introduce _any_ user-facing change?
These changes should NOT affect anyone who is not trying build or run bash scripts from a Windows SHELL environment.
A `cygwin` user that prefers to call the `Windows` version of the spark scripts (`spark-shell.cmd`, etc) from a ` bash` session would need to unset the `SHELL` environment variable for it to work as before, e.g.
```bash
SHELL= spark-shell.cmd
```
### How was this patch tested?
Manual tests were performed to verify both changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk commented on pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1274813553
> (@philwalk rebasing it would retrigger the Github Actions jobs)
I did the following, hope it was correct:
```bash
git fetch upstream
git rebase upstream/master
git pull
git commit -m 'rebase to trigger build'
git push
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1273155534
Thanks for the contribution. Would you mind checking https://github.com/apache/spark/pull/38167/checks?check_run_id=8783733198 and https://spark.apache.org/contributing.html? e.g., let's file a JIRA and link it to the PR title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk commented on pull request #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1273797450
The following 2 JIRA issue were created. Both are fixed by this PR. They are both linked to this PR.
- Bug SPARK-40739 "sbt packageBin" fails in cygwin or other windows bash session
- Bug SPARK-40738 spark-shell fails with "bad array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk commented on pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1276679559
I can delete the fork and start over, but is there a shortcut that accomplishes the same thing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk commented on pull request #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1273489251
> Would you mind checking https://github.com/apache/spark/pull/38167/checks?check_run_id=8783733198
Two suggestions are provided:
Enable Github Actions:
My fork appears to be configured to allow actions, although I'm not sure. Here's what I see:
Actions permissions
- Any action or reusable workflow can be used, regardless of who authored it or where it is defined.
Workflow permissions
- Workflows have read and write permissions in the repository for all scopes.
Allow Github Actions to create and approve pull requests
The second suggestion is this:
```bash
git fetch upstream
git rebase upstream/master
git push origin YOUR_BRANCH --force
```
I just did so, although it didn't fix the problem.
UPDATE: I found the screen for enabling workflows, so we should be okay to re-run the failed check now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk commented on pull request #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1273490984
> https://spark.apache.org/contributing.html? e.g., let's file a JIRA and link it to the PR title.
I'm looking into it now on the JIRA website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1276284019
I think this is messed up now, not sure how as your approach seems OK (though you would have had to force push)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1274034934
(@philwalk rebasing it would retrigger the Github Actions jobs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] philwalk closed pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
philwalk closed pull request #38167: [SPARK-40739][SPARK-40738] Fix problems that affect windows shell environments (cygwin/msys2/mingw)
URL: https://github.com/apache/spark/pull/38167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38167: fix problems that affect windows shell environments (cygwin/msys2/mingw)
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38167:
URL: https://github.com/apache/spark/pull/38167#issuecomment-1272698437
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org