You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "edwardcapriolo (via GitHub)" <gi...@apache.org> on 2026/01/09 15:53:08 UTC
[PR] LIVY-1030 cleanup of context and clients [incubator-livy]
edwardcapriolo opened a new pull request, #506:
URL: https://github.com/apache/incubator-livy/pull/506
## What changes were proposed in this pull request?
An NPE, some unused variables, and a more clear design for the client factories
https://issues.apache.org/jira/browse/LIVY-1030
## How was this patch tested?
Unit tests.
Please review https://livy.incubator.apache.org/community/ before opening a pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by larry mccay <la...@gmail.com>.
Again, this is a community decision and not one to make at a pull request
level. Bring it to the dev@ list and drive the DISCUSS, change the PR to
pass in current project state or wait for the community to bring it up.
It's not getting merged while failing test runs on currently supported Java
versions and we aren't dropping support for a clean up pull request.
On Fri, Jan 16, 2026, 6:24 PM edwardcapriolo (via GitHub) <gi...@apache.org>
wrote:
>
> edwardcapriolo commented on PR #506:
> URL:
> https://github.com/apache/incubator-livy/pull/506#issuecomment-3762197767
>
> As it relates to Java 8. I used to be a system admin, and I am all
> about backwards compatibility. But for realism sake: Spark is at 3.5 (17)
> and 4.X. I find it really hard to believe someone would pick up livy trunk
> today, and say "I really want to run new shiny living on spark 2.3 and
> hadoop 2.4 on java 8". Likewise I cant imagine anyone using livy-0.7 on
> spark2 and then saying "i need this critical bug fix in trunk" I would say
> branch off an old one.
>
>
> --
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2709338393
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
@edwardcapriolo - is this 'ignored' somehow used? Is it not scoped to this finally block only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2709405826
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
My IDE will mark unused variables as yellow. You have a few options two of which:
1. you could add a log statement
2. you can name the variable 'ignored'
I strive to make as many files as possible "fully green". Here i took option 2
<img width="1472" height="1137" alt="Screenshot From 2026-01-20 12-36-56" src="https://github.com/user-attachments/assets/8275201d-9ba5-4fa6-9b87-cb7c7a741ce9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3762197767
As it relates to Java 8. I used to be a system admin, and I am all about backwards compatibility. But for realism sake: Spark is at 3.5 (17) and 4.X. I find it really hard to believe someone would pick up livy trunk today, and say "I really want to run new shiny living on spark 2.3 and hadoop 2.4 on java 8". Likewise I cant imagine anyone using livy-0.7 on spark2 and then saying "i need this critical bug fix in trunk" I would say branch off an old one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3780772458
@edwardcapriolo - thanks for your contribution. This has been merged and will be included in the next release off of master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3761059850
> @lmccay Also
>
> the errors:
>
> at org.apache.livy.thriftserver.LivyThriftSessionManager.$anonfun$openSession$2(LivyThriftSessionManager.scala:269) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) ... Cause: java.io.IOException: RSCClient instance stopped. at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:245) at org.apache.livy.rsc.RSCClient.connectionError(RSCClient.java:155) at org.apache.livy.rsc.RSCClient.access$300(RSCClient.java:51) at org.apache.livy.rsc.RSCClient$1.onFailure(RSCClient.java:95) at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503)
>
> I think this entire suite is flakey, Just my opinion but they fail sometimes when i run master. They fail randomly quite often.
We should revisit those then. If there is no JIRA then go ahead and file one and target 0.10.0 release. NOTE: if we drop java 8 altogether at that time, it may be a 1.0.0.
There are plenty of PRs that aren't failing these, so trying a rerun should get a pass at some point.
I've rerun a couple on your PRs, not sure if it includes this one, and they were still failing.
I'm going to be out of pocket for a bit - maybe will make it back later this evening but heading out now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2701527254
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -181,13 +174,16 @@ private static ChildProcess startDriver(final RSCConf conf, Promise<?> promise)
}
Utils.checkState(rscJars.isDirectory(),
- "Cannot find rsc jars directory under LIVY_HOME.");
+ "Cannot find rsc jars directory: " + rscJars.getAbsolutePath());
allJars.add(rscJars);
List<String> jars = new ArrayList<>();
for (File dir : allJars) {
- for (File f : dir.listFiles()) {
- jars.add(f.getAbsolutePath());
+ File [] list = dir.listFiles();
Review Comment:
Identified as possible NullPointerException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2710333660
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
I think its good practice. Ignore return values is usually a findbugs warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3765457919
Additionally I have created https://issues.apache.org/jira/browse/LIVY-1033. I do not believe I ever had R on my system and my command line invocations never failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2710345796
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
Remove it or replace it with logging which will actually do something. It is not an existing practice in that file to add local vars that have no value and that may look like a bug. I can see a log message at least adding value in case it failed to delete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3780193254
@lmccay Thank you. Since we are PR BFFs may you look at this one :) https://github.com/apache/incubator-livy/pull/507 it should be a slam dunk. Famous last words
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2701527254
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -181,13 +174,16 @@ private static ChildProcess startDriver(final RSCConf conf, Promise<?> promise)
}
Utils.checkState(rscJars.isDirectory(),
- "Cannot find rsc jars directory under LIVY_HOME.");
+ "Cannot find rsc jars directory: " + rscJars.getAbsolutePath());
allJars.add(rscJars);
List<String> jars = new ArrayList<>();
for (File dir : allJars) {
- for (File f : dir.listFiles()) {
- jars.add(f.getAbsolutePath());
+ File [] list = dir.listFiles();
Review Comment:
Identified as possible NullPointerException which is fairly likely with the diverse expansive classpaths of Livy, Spark, and Hadoop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3760897567
@lmccay Also
the errors:
at org.apache.livy.thriftserver.LivyThriftSessionManager.$anonfun$openSession$2(LivyThriftSessionManager.scala:269)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
...
Cause: java.io.IOException: RSCClient instance stopped.
at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:245)
at org.apache.livy.rsc.RSCClient.connectionError(RSCClient.java:155)
at org.apache.livy.rsc.RSCClient.access$300(RSCClient.java:51)
at org.apache.livy.rsc.RSCClient$1.onFailure(RSCClient.java:95)
at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503)
I think this entire suite is flakey, Just my opinion but they fail sometimes when i run master. They fail randomly quite often.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2709405826
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
My IDE will mark unused variables as yellow. You have a few options two of which:
1. you could add a log statement
2. you can name the variable 'ignored'
I strive to make as many files as possible "fully green". Here i took option 2. Explicitly ignored not ignored because I was unaware of a return.
<img width="1472" height="1137" alt="Screenshot From 2026-01-20 12-36-56" src="https://github.com/user-attachments/assets/8275201d-9ba5-4fa6-9b87-cb7c7a741ce9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2710333660
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
I think its good practice. Ignore return values is usually a findbugs warning. Like "mkdir() returns 1 when it fails to make the directory."
https://softdevbuilttolast.wordpress.com/2010/03/23/findbugs-help-create-robust-code-with-the-checkreturnvalue-annotation/
The yellow in my IDE is what called my attention to the initial real null pointer issue in the loop that processes the classpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3760833009
@lmccay java 8 is dead. Hadoop removed. Lets remove the java8 builder. Spark 3.4 has minimum java 17,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3764313731
@lmccay You are right. I found an issue I created.
```java
//Note. Your compiler or IDE may identify this method as unused
//tests fail without it
public void handle(ChannelHandlerContext ctx, RemoteDriverAddress msg) {
```
Note to other developers: The test Running org.apache.livy.rsc.TestSparkClient often creates a fork-bomb situation on my computer by rapidly launching multiple sparks on my system creating memory pressure. But they do run, I moved to a computer with 16GB of ram.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3780775369
> @edwardcapriolo - thanks for your contribution. This has been merged and will be included in the next release off of master!
Thanks for your help as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay merged PR #506:
URL: https://github.com/apache/incubator-livy/pull/506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2710333660
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
I think its good practice. Ignore return values is usually a findbugs warning. Like "mkdir() returns 1 when it fails to make the directory."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2712526390
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
Yes. I agree that is the right coarse of action add a logging message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3760888624
> @lmccay Hadoop 3.5.0 is now min java 17. Spark 3.4 has minimum java 17. Lets remove the java8 branch builder and move this project forward. It isn't possible to fix things into the "deep past"
This requires a larger discussion than here on a specific pull request. It is a discussion that would need to be had on the plans for the next release. I am supportive of it, in general, but again, it needs discussion on the dev@ list. Feel free to start the DISCUSS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3764091267
> As it relates to Java 8. I used to be a system admin, and I am all about backwards compatibility. But for realism sake: Spark is at 3.5 (17) and 4.X. I find it really hard to believe someone would pick up livy trunk today, and say "I really want to run new shiny living on spark 2.3 and hadoop 2.4 on java 8". Likewise I cant imagine anyone using livy-0.7 on spark2 and then saying "i need this critical bug fix in trunk" I would say branch off an old one.
Again, this is a community decision and not one to make at a pull request level. Bring it to the dev@ list and drive the DISCUSS, change the PR to pass in current project state or wait for the community to bring it up.
It's not getting merged while failing test runs on currently supported Java versions and we aren't dropping support for a java version for a clean up pull request failing tests. There are aspects to consider in dropping the version of java than just dropping support that need to be discussed as well. We will need to determine the near term policy on java 17+ syntax and the effect on backports while previous lines are still technically supported and may need at least security patches. These are the sorts of things that make it a larger discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3764194661
> @lmccay
>
> > > It's not getting merged while failing test runs on currently supported Java versions and we aren't dropping support for a java version for a clean up pull request failing tests.
>
> Of course, I wouldn't ask you to merge something that is breaking. Have you looked at the PR? All i did was change a method that returns a null to a method that returns Option.
>
> Did I actually produce bad code or is this simply flaky builds? java8 has optional support, so i don't see why it would break?
>
> I observe these test fail on MASTER
>
> ```shell
> edward@fedora:~/incubator-livy$ git log | head -10
> commit 284467b98b8f086cb3cf9a6813873141bced5dc2
> Author: Edward Capriolo <ed...@gmail.com>
> Date: Fri Jan 16 10:24:14 2026 -0500
>
> LIVY-1029 modernize versions (#505)
>
>
> [INFO] --- surefire:2.22.1:test (default-test) @ livy-rsc ---
> [WARNING] Parameter 'systemProperties' is deprecated: Use systemPropertyVariables instead.
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.livy.rsc.TestJobHandle
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.082 s - in org.apache.livy.rsc.TestJobHandle
> [INFO] Running org.apache.livy.rsc.TestSparkClient
> [ERROR] Tests run: 19, Failures: 0, Errors: 18, Skipped: 0, Time elapsed: 28.672 s <<< FAILURE! - in org.apache.livy.rsc.TestSparkClient
> [ERROR] testSyncRpc(org.apache.livy.rsc.TestSparkClient) Time elapsed: 5.65 s <<< ERROR!
> java.util.concurrent.ExecutionException: java.io.IOException: RSCClient instance stopped.
> at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:593)
> at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:580)
> at org.apache.livy.rsc.TestSparkClient.testSyncRpc(TestSparkClient.java:202)
> Caused by: java.io.IOException: RSCClient instance stopped.
> ```
>
> It isn't a reason to block my commit because the tests don't run at all :). This is the exact reason I reached out to the incubator-ml to come on as a committer, because I think maybe no one is aware that the builds don't work, or they dont seem to care.
Well, I built master and those tests pass. I build with your patch applied and it fails. Deterministically, with the error that you show above. Built the same way with the docker build. Have you tried building master without your change? What run is the above output from?
I do see other pull requests with the builds passing.
Note that the build command that I am using is the following which is also being fixed in the docs as the proper build command [1]:
docker run --rm -it -v $(pwd):/workspace -v $HOME/.m2:/root/.m2 livy-ci mvn clean package -Pspark3 -Pscala-2.12
I am leaving your community complaints aside, as we try and work through your contirbutions for the value that they can provide.
1. https://issues.apache.org/jira/browse/LIVY-1031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2701557352
##########
api/src/main/java/org/apache/livy/LivyClientBuilder.java:
##########
@@ -134,20 +131,24 @@ public LivyClient build() {
LivyClient client = null;
if (CLIENT_FACTORIES.isEmpty()) {
- throw new IllegalStateException("No LivyClientFactory implementation was found.");
+ throw new IllegalStateException("No LivyClientFactory implementations were found.");
}
for (LivyClientFactory factory : CLIENT_FACTORIES) {
try {
- client = factory.createClient(uri, config);
- } catch (Exception e) {
- if (!(e instanceof RuntimeException)) {
- e = new RuntimeException(e);
+ Optional<LivyClient> found = factory.createClient(uri, config);
Review Comment:
This is still a slightly "funky" pattern. It might be better to make an interface with two methods. One advertising what protocols the Client supports, then the other returning the Client. I went with returning Optional instead of sometimes returning null. You might want to take it or leave it. If people hate it the other cleanups are still nice.
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -181,13 +174,16 @@ private static ChildProcess startDriver(final RSCConf conf, Promise<?> promise)
}
Utils.checkState(rscJars.isDirectory(),
- "Cannot find rsc jars directory under LIVY_HOME.");
+ "Cannot find rsc jars directory: " + rscJars.getAbsolutePath());
Review Comment:
Motivation: Better to tell people where it is looking exactly instead of telling them where Livy root is.
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
}
};
Thread thread = new Thread(wrappedTask);
thread.setDaemon(true);
thread.setName("ContextLauncher-" + childId);
- thread.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
- @Override
- public void uncaughtException(Thread t, Throwable e) {
- LOG.warn("Child task threw exception.", e);
- fail(e);
- }
+ thread.setUncaughtExceptionHandler((t, e) -> {
+ LOG.warn("Child task threw exception.", e);
+ fail(e);
});
thread.start();
return thread;
}
}
+ public RSCConf getConf() {
Review Comment:
Compile claimed RSConf is unused. I didn't want to disrupt the constructor so added a getter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2709409402
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
<img width="943" height="1108" alt="Screenshot From 2026-01-20 12-38-14" src="https://github.com/user-attachments/assets/def83479-a629-4973-9595-7a071c7725d0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3761063985
Oh, this is indeed the one that I retried a couple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3764214416
@lmccay
[WARNING] Parameter 'systemProperties' is deprecated: Use systemPropertyVariables instead.
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.livy.rsc.TestJobHandle
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.387 s - in org.apache.livy.rsc.TestJobHandle
[INFO] Running org.apache.livy.rsc.TestSparkClient
^Cedward@fedora:~/incubator-livy$ export JAVA_HOME=/usr/lib/jvm/java-17-temurin-jdk/
Clean checkout java 17 first RSC method hands then fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3764158719
@lmccay
>> It's not getting merged while failing test runs on currently supported Java versions and we aren't dropping support for a java version for a clean up pull request failing tests.
Of course, I wouldn't ask you to merge something that is breaking. Have you looked at the PR? All i did was change a method that returns a null to a method that returns Option.
Did I actually produce bad code or is this simply flaky builds?
Jjava8 has optional support so i dont see why it would break?
I observe these test fail on MASTER
```shell
edward@fedora:~/incubator-livy$ git log | head -10
commit 284467b98b8f086cb3cf9a6813873141bced5dc2
Author: Edward Capriolo <ed...@gmail.com>
Date: Fri Jan 16 10:24:14 2026 -0500
LIVY-1029 modernize versions (#505)
[INFO] --- surefire:2.22.1:test (default-test) @ livy-rsc ---
[WARNING] Parameter 'systemProperties' is deprecated: Use systemPropertyVariables instead.
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.livy.rsc.TestJobHandle
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.082 s - in org.apache.livy.rsc.TestJobHandle
[INFO] Running org.apache.livy.rsc.TestSparkClient
[ERROR] Tests run: 19, Failures: 0, Errors: 18, Skipped: 0, Time elapsed: 28.672 s <<< FAILURE! - in org.apache.livy.rsc.TestSparkClient
[ERROR] testSyncRpc(org.apache.livy.rsc.TestSparkClient) Time elapsed: 5.65 s <<< ERROR!
java.util.concurrent.ExecutionException: java.io.IOException: RSCClient instance stopped.
at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:593)
at org.apache.livy.rsc.TestSparkClient.runTest(TestSparkClient.java:580)
at org.apache.livy.rsc.TestSparkClient.testSyncRpc(TestSparkClient.java:202)
Caused by: java.io.IOException: RSCClient instance stopped.
```
It isnt a reason to block my commit because the tests dont run at all :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "edwardcapriolo (via GitHub)" <gi...@apache.org>.
edwardcapriolo commented on PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#issuecomment-3780364952
> @edwardcapriolo - did you just add the bump versions commit to this with this last iteration? Why mix them?
I cleaned this up. My bad. Switching branches a often. 2x check for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-1030 cleanup of context and clients [incubator-livy]
Posted by "lmccay (via GitHub)" <gi...@apache.org>.
lmccay commented on code in PR #506:
URL: https://github.com/apache/incubator-livy/pull/506#discussion_r2710270271
##########
rsc/src/main/java/org/apache/livy/rsc/ContextLauncher.java:
##########
@@ -436,36 +422,34 @@ public void detach() {
try {
monitor.join(conf.getTimeAsMs(CLIENT_SHUTDOWN_TIMEOUT));
} catch (InterruptedException ie) {
- LOG.debug("Interrupted before driver thread was finished.");
+ LOG.debug("Interrupted before driver thread was finished.", ie);
}
}
private Thread monitor(final Runnable task, int childId) {
- Runnable wrappedTask = new Runnable() {
- @Override
- public void run() {
- try {
- task.run();
- } finally {
- confFile.delete();
- }
+ Runnable wrappedTask = () -> {
+ try {
+ task.run();
+ } finally {
+ boolean ignored = confFile.delete();
Review Comment:
Please remove that. I spent time trying to figure out why it was necessary. We don't need your IDE to be fully green here. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org