You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/02/21 12:56:42 UTC
[GitHub] [incubator-kyuubi] yaooqinn opened a new pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
yaooqinn opened a new pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The original idea of SPARK_HADOOP_VERSION is used to concat spark release names only, now we need to remove it as
- SPARK_HADOOP_VERSION is misunderstood by developers and misused somewhere
- multi-engine support now
- the release names of spark(or something else) are very easy to get through code with different environments, prod/test/dev
- A `mvn` job is bundled with `bin/load-kyuubi-env.sh` which is truly worrisome
- SPARK_HADOOP_VERSION on spark side is broken already for spark 3.2 which actually bundled with hadoop 3.3 see https://github.com/apache/spark-website/pull/361#discussion_r730716668
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811532833
##########
File path: pom.xml
##########
@@ -1901,20 +1900,6 @@
</properties>
</profile>
- <profile>
Review comment:
when we update spark
before: we update 2 parameters `spark.version` and `hadoop.binary.version`, and we need to ensure in our mind that the `spark.archive.name` is also need to update.
now: we update 1 parameter `spark.version` only or sometimes 2 if `spark.archive.name` changes its static pattern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811203046
##########
File path: bin/load-kyuubi-env.sh
##########
@@ -109,7 +88,6 @@ if [ $silent -eq 0 ]; then
echo "JAVA_HOME: ${JAVA_HOME}"
echo "KYUUBI_HOME: ${KYUUBI_HOME}"
- echo "KYUUBI_VERSION: ${KYUUBI_VERSION}"
Review comment:
shall we keep the kyuubi version here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811514277
##########
File path: pom.xml
##########
@@ -1901,20 +1900,6 @@
</properties>
</profile>
- <profile>
Review comment:
I'm not sure if it's good to remove the Spark Hadoop version profile. The Spark master branch has already moved to Hadoop 3.3.1. So if we upgrate Spark to 3.3 we still need the profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811534791
##########
File path: pom.xml
##########
@@ -1901,20 +1900,6 @@
</properties>
</profile>
- <profile>
Review comment:
I see, then I'm fine with this change. We have already use `spark.archive.name` in ga workflow, so the Spark Hadoop version can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#issuecomment-1047390211
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811514277
##########
File path: pom.xml
##########
@@ -1901,20 +1900,6 @@
</properties>
</profile>
- <profile>
Review comment:
I'm not sure if it's good to remove the Spark Hadoop version profile. The Spark master branch has already moved to Hadoop 3.3.1. So if we upgrade Spark to 3.3 we still need the profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811207382
##########
File path: bin/load-kyuubi-env.sh
##########
@@ -109,7 +88,6 @@ if [ $silent -eq 0 ]; then
echo "JAVA_HOME: ${JAVA_HOME}"
echo "KYUUBI_HOME: ${KYUUBI_HOME}"
- echo "KYUUBI_VERSION: ${KYUUBI_VERSION}"
Review comment:
it will be printed in stdout asap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #1950: Remove ambiguous SPARK_HADOOP_VERSION
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #1950:
URL: https://github.com/apache/incubator-kyuubi/pull/1950#discussion_r811206503
##########
File path: bin/load-kyuubi-env.sh
##########
@@ -109,7 +88,6 @@ if [ $silent -eq 0 ]; then
echo "JAVA_HOME: ${JAVA_HOME}"
echo "KYUUBI_HOME: ${KYUUBI_HOME}"
- echo "KYUUBI_VERSION: ${KYUUBI_VERSION}"
Review comment:
it will be print in log ASAP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org