You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/12/13 14:48:50 UTC
[PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
ArnavBalyan opened a new pull request, #503:
URL: https://github.com/apache/incubator-livy/pull/503
## What changes were proposed in this pull request?
- Ensures the CWD for interactive sessions is set to Spark root directory for data files to be available.
- Deprecated unused Spark 1 code.
- Adds UTs to ensure files are available when set for interactive sessions.
## How was this patch tested?
- UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3947860056
I have now merged the PR to enable both unit and integration tests for PRs, could you please rebase your PR? If the unit tests are successful, we could merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3851326287
cc @gyogal gentle reminder thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3982222867
Thanks @gyogal I'll raise a new PR with all changes fixed closing this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3783896024
Hi @gyogal I tested the flow again, the PySpark issue may be due to missing local pyspark, I'm able to run locally with pyspark available in the env. The test is using the PythonInterpreter API, which is not available on the livy-integration-test. Making it available there would require a refactor, and some wiring up for pyspark thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan closed pull request #503: [LIVY-789] Ensure Spark data files are available in interactive session
URL: https://github.com/apache/incubator-livy/pull/503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3693040594
cc @gyogal gentle reminder when you have time thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3697620729
Hi @ArnavBalyan , thanks for submitting a PR for this issue. I think the change itself looks good, but I am not able to run the test successfully. This is what I am getting on my machine:
```
- should open files localized via spark.files by relative path *** FAILED *** (672 milliseconds)
ExecuteAborted(Traceback (most recent call last):
File ".../pr-review/repl/scala-2.12/target/tmp/2817641729331086602", line 664, in <module>
sys.exit(main())
^^^^^^
File ".../pr-review/repl/scala-2.12/target/tmp/2817641729331086602", line 523, in main
exec('from pyspark.sql import HiveContext', global_dict)
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyspark') did not equal ExecuteSuccess(JObject(List((text/plain,JString('hello-files'))))) (PythonInterpreterSpec.scala:300)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#discussion_r2870510908
##########
server/src/main/scala/org/apache/livy/utils/SparkYarnApp.scala:
##########
@@ -155,8 +153,9 @@ class SparkYarnApp private[utils] (
// We cannot kill the YARN app without the app id.
// There's a chance the YARN app hasn't been submitted during a livy-server failure.
// We don't want a stuck session that can't be deleted. Emit a warning and move on.
- case _: TimeoutException | _: InterruptedException =>
+ case e @ (_: InterruptedException | NonFatal(_)) =>
Review Comment:
Thanks for working on this! The tests now look good, but I am wondering why the LIVY-1016 commit is showing in the PR (this file and the one below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3652722223
cc @gyogal @lmccay thanks! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3755720603
> Hi @ArnavBalyan , thanks for submitting a PR for this issue. I think the change itself looks good, but I am not able to run the test successfully. This is what I am getting on my machine:
>
> ```
> - should open files localized via spark.files by relative path *** FAILED *** (672 milliseconds)
> ExecuteAborted(Traceback (most recent call last):
> File ".../pr-review/repl/scala-2.12/target/tmp/2817641729331086602", line 664, in <module>
> sys.exit(main())
> ^^^^^^
> File ".../pr-review/repl/scala-2.12/target/tmp/2817641729331086602", line 523, in main
> exec('from pyspark.sql import HiveContext', global_dict)
> File "<string>", line 1, in <module>
> ModuleNotFoundError: No module named 'pyspark') did not equal ExecuteSuccess(JObject(List((text/plain,JString('hello-files'))))) (PythonInterpreterSpec.scala:300)
> ```
>
> Edit: It may be possible to move this test to `InteractiveIT` in `livy-integration-test`, and then the `spark.livy.forceSparkFilesTest` part would not be needed since integration tests already run with `LIVY_TEST=false`.
Hey Gyorgy, thanks for raising this let me check I was able to run the test a while back, `livy-integration-test` is a great idea, will test it out and get back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-789] Ensure Spark data files are available in interactive session [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #503:
URL: https://github.com/apache/incubator-livy/pull/503#issuecomment-3917501766
@ArnavBalyan Could you please try running the unit test for this PR by rebasing to [this commit](https://github.com/apache/incubator-livy/pull/513/changes) or by adding a similar change to this PR? If the unit test passes, this could be merged but for some reason the newly added test fails for me when running locally (with no pyspark on the PATH).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org