You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/10/24 09:20:24 UTC
[PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
ArnavBalyan opened a new pull request, #489:
URL: https://github.com/apache/incubator-livy/pull/489
## What changes were proposed in this pull request?
- Spark 2.x is at end of life and no longer supported, newer Spark versions (Spark 4) are now available.
- Deprecate Spark 2.x from Livy
## How was this patch tested?
- CI and local builds are successful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3454106861
cc @gyogal gentle reminder thanks! 😀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3444566740
Thank you for your PR @ArnavBalyan! This could be an important step of cleaning up unneeded features in an upcoming release. The change is quite big, so it will take some time to review. In the meantime, can I ask why you chose Spark version 3.3.4 in the main `pom.xml` and in `README.md`? The latest version should work with 3.5.6, but Spark 3.5.7 is already available as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3508957784
Hi @ArnavBalyan , the main Livy version has been updated to 0.10-incubating-SNAPSHOT, however this caused some conflicts in your PR. Could you please resolve them? Once that is done, I think this could be merged into version 0.10-incubating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3486239496
Hi @gyogal thanks for taking a look! Makes perfect sense, will wait until we do the release. Let me know if you need support with release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3442209917
cc @gyogal @lmccay could you ptal thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3530456340
Thank you for your contribution @ArnavBalyan !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal merged PR #489:
URL: https://github.com/apache/incubator-livy/pull/489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3444753257
> For Spark, looks like `3.3.4` was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look
Oh right, it makes sense now, thanks! In #478 it has already been changed to version 3 in the `spark3` profile, so I think that should be OK. Alternatively, it could be done in a follow-up change later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3527359027
Thanks @gyogal! Will rebase it shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3444656331
> Thank you for your PR @ArnavBalyan! This could be an important step of cleaning up unneeded features in an upcoming release. The change is quite big, so it will take some time to review. In the meantime, can I ask why you chose Spark version 3.3.4 in the main `pom.xml` and in `README.md`? The latest version should work with 3.5.6, but Spark 3.5.7 is already available as well.
Hi @gyogal thanks so much, really appreciate it. For Spark, looks like `3.3.4` was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3448438244
> > For Spark, looks like `3.3.4` was the latest version that shipped hadoop 2 binaries. Moving to 3.5.7 may need us to bump to hadoop 3, was a bit unsure if this would have any side effects since this was getting a considerably large change. Let me know if we are comfortable directly moving to hadoop 3 happy to update. Thanks for taking a look
>
> Oh right, it makes sense now, thanks! In #478 it has already been changed to version 3 in the `spark3` profile, so I think that should be OK. Alternatively, it could be done in a follow-up change later on.
Thanks @gyogal I can bump it up as a follow up change if it looks good to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1019] Deprecate Spark 2 in livy [incubator-livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on PR #489:
URL: https://github.com/apache/incubator-livy/pull/489#issuecomment-3481723877
@ArnavBalyan , I think both this and the Python 3 commit could be merged after `branch-0.9` is created. That would mean that the upcoming 0.9 release would still retain support, but the cleanup work can be started in preparation for the release afterwards. Please let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org