You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "ksumit (via GitHub)" <gi...@apache.org> on 2023/03/14 16:38:35 UTC
[GitHub] [incubator-livy] ksumit opened a new pull request, #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
ksumit opened a new pull request, #394:
URL: https://github.com/apache/incubator-livy/pull/394
## What changes were proposed in this pull request?
Added support for specific flags to support hadoop/spark/scala version specific builds.
## How was this patch tested?
Ran `mvn install` with the new flags to confirm they work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] ayushtkn commented on a diff in pull request #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #394:
URL: https://github.com/apache/incubator-livy/pull/394#discussion_r1147898935
##########
README.md:
##########
@@ -88,3 +88,13 @@ between different Spark versions.
The Livy package itself does not contain a Spark distribution. It will work with any supported
version of Spark without needing to rebuild.
+
+### Build Profiles
+
+| Flag | Purpose |
+|-------------|--------------------------------------------------------------------|
+| -Phadoop2 | Choose Hadoop2 based build dependencies (default configuration) |
+| -Pspark2 | Choose Spark 2.x based build dependencies (default configuration) |
+| -Pscala-2.11 | Choose Scala 2.11 based build dependencies (default configuration) |
+| -Pscala-2.12 | Choose scala 2.12 based build dependencies |
+| -Pspark3 | Choose Spark 3.x based build dependencies |
Review Comment:
This sounds tough to do and very hard coded, Do we have an option like -Dhadoop.version=3.3.5 and have a way to provide them at runtime, if we can do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] ksumit closed pull request #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
Posted by "ksumit (via GitHub)" <gi...@apache.org>.
ksumit closed pull request #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
URL: https://github.com/apache/incubator-livy/pull/394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] ayushtkn commented on a diff in pull request #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #394:
URL: https://github.com/apache/incubator-livy/pull/394#discussion_r1147898935
##########
README.md:
##########
@@ -88,3 +88,13 @@ between different Spark versions.
The Livy package itself does not contain a Spark distribution. It will work with any supported
version of Spark without needing to rebuild.
+
+### Build Profiles
+
+| Flag | Purpose |
+|-------------|--------------------------------------------------------------------|
+| -Phadoop2 | Choose Hadoop2 based build dependencies (default configuration) |
+| -Pspark2 | Choose Spark 2.x based build dependencies (default configuration) |
+| -Pscala-2.11 | Choose Scala 2.11 based build dependencies (default configuration) |
+| -Pscala-2.12 | Choose scala 2.12 based build dependencies |
+| -Pspark3 | Choose Spark 3.x based build dependencies |
Review Comment:
This sounds tough to do and very hard coded, Do we have an option like -Dhadoop.version=3.3.5 and have a way to provide them at runtime, if we can do that I think we are sorted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] ksumit commented on a diff in pull request #394: [LIVY-593] Add flags to support hadoop, spark and scala version specific builds
Posted by "ksumit (via GitHub)" <gi...@apache.org>.
ksumit commented on code in PR #394:
URL: https://github.com/apache/incubator-livy/pull/394#discussion_r1149793135
##########
README.md:
##########
@@ -88,3 +88,13 @@ between different Spark versions.
The Livy package itself does not contain a Spark distribution. It will work with any supported
version of Spark without needing to rebuild.
+
+### Build Profiles
+
+| Flag | Purpose |
+|-------------|--------------------------------------------------------------------|
+| -Phadoop2 | Choose Hadoop2 based build dependencies (default configuration) |
+| -Pspark2 | Choose Spark 2.x based build dependencies (default configuration) |
+| -Pscala-2.11 | Choose Scala 2.11 based build dependencies (default configuration) |
+| -Pscala-2.12 | Choose scala 2.12 based build dependencies |
+| -Pspark3 | Choose Spark 3.x based build dependencies |
Review Comment:
It is hardcoded intentionally, getting different versions of hadoop, hive and spark to play with a version of livy and making it work across their dependency versions is a hard problem. you would notice that the functionality to pass and override [hadoop](https://github.com/apache/incubator-livy/blob/master/pom.xml#L82), [scala](https://github.com/apache/incubator-livy/blob/master/pom.xml#LL109-L111C57), [spark](https://github.com/apache/incubator-livy/blob/master/pom.xml#LL86-L86C63) and other versions is already available but we still need these profiles to resolve version incompatibilities in dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org