You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2023/04/11 08:48:58 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #40738: [BUILD] Test maven 3.9.1
LuciferYang opened a new pull request, #40738:
URL: https://github.com/apache/spark/pull/40738
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bowenliang123 commented on a diff in pull request #40738: [BUILD] Test maven 3.9.1
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #40738:
URL: https://github.com/apache/spark/pull/40738#discussion_r1164910908
##########
build/mvn:
##########
@@ -36,7 +36,7 @@ _DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Preserve the calling directory
_CALLING_DIR="$(pwd)"
# Options used during compilation
-_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m"
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m -Dmaven.resolver.transport=wagon"
Review Comment:
Hi, a kind ping from Kyuubi, we reverted to using Maven 3.8.x in Kyuubi for CI builds and local usage, to reduce side effects and easier to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1522123005
Sorry for missing pings, @LuciferYang . I'm back from my vacation.
I'm also +1 for the decision (waiting for new better version). Thank you for closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #40738:
URL: https://github.com/apache/spark/pull/40738#discussion_r1164911561
##########
build/mvn:
##########
@@ -36,7 +36,7 @@ _DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Preserve the calling directory
_CALLING_DIR="$(pwd)"
# Options used during compilation
-_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m"
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m -Dmaven.resolver.transport=wagon"
Review Comment:
Thanks for your sync @bowenliang123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #40738:
URL: https://github.com/apache/spark/pull/40738#discussion_r1163755153
##########
build/mvn:
##########
@@ -36,7 +36,7 @@ _DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Preserve the calling directory
_CALLING_DIR="$(pwd)"
# Options used during compilation
-_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m"
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m -Dmaven.resolver.transport=wagon"
Review Comment:
try to specify `-Dmaven.resolver.transport=wagon` due 3.8.x use `wagon` as default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1519081149
I mean, is it worth updating if 3.9.1 introduces small issues that we have to work around? or just wait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1511614880
@dongjoon-hyun @steveloughran Adding `-Dmaven.resolver.transport=wagon` seems useful, as recent Java 11&17 Maven build have all finished within 1 hour and there have been no more timeouts. Let me check for two more days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
URL: https://github.com/apache/spark/pull/40738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1503380076
The property `aether.connector.connectTimeout` seems nogt working, and there are still maven compilation performance issues, need more investigation ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1521522776
me, I'm happy with whatever suits, just noting that 3.9.1 was unwelcome trouble for me as it ended up hurting the 3.3.5 release process -not the docker builds, which were still on a lower version, but my local dev builds blew up. joy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1521073578
@steveloughran Do you have any other questions? I am planning to close this PR and test Apache Maven 3.9.2+ in the future to check if it is possible not to add `-Dmaven.resolver.transport=wagon`. Apache Spark can continue to use 3.8.7 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #40738:
URL: https://github.com/apache/spark/pull/40738#discussion_r1163755153
##########
build/mvn:
##########
@@ -36,7 +36,7 @@ _DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Preserve the calling directory
_CALLING_DIR="$(pwd)"
# Options used during compilation
-_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m"
+_COMPILE_JVM_OPTS="-Xss128m -Xmx4g -XX:ReservedCodeCacheSize=128m -Dmaven.resolver.transport=wagon"
Review Comment:
try to specify `-Dmaven.resolver.transport=wagon`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bowenliang123 commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1503444531
Agree. Same on kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1521616040
Thanks for your attention to this pr @bowenliang123 @dongjoon-hyun @srowen @steveloughran, close first ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1503770505
Thank you all.
FYI, @steveloughran . Apache Maven 3.9.1 seems to have issues still according to the above discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #40738:
URL: https://github.com/apache/spark/pull/40738#discussion_r1163592312
##########
connector/protobuf/pom.xml:
##########
@@ -92,7 +92,7 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
- <shadeTestJar>true</shadeTestJar>
+ <shadeTestJar>false</shadeTestJar>
Review Comment:
[22c0173](https://github.com/apache/spark/pull/40738/commits/22c0173793ae5b978103bb2050e66bd72336d86f) hang when maven shade to downloads some non-existent tests. jar and set it to false to try again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] steveloughran commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1505253649
noted. interestingly for cutting hadoop releases i have had to turn off connection pooling so as to guarantee reliable builds (docker container having to download **/*.jar from maven central is too brittle otherwise). that's even older versions.
* spark &c with fixed versions should stay on the good ones
* if the latest version of the SBOR is happy on recent maven builds then we can reinstate in hadoop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bowenliang123 commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1521624076
Thanks for the ping. And welcome to cc me when Spark has the next attempt to use maven 3.9.x/4.x.
I was pessimistic that whether Maven will change the default timeout as it's a default config (`aether.connector.connectTimeout=10000`) of Maven Artifact Resolver not in Maven itself. And as for Maven, it has changed to Artifact Resolver as the default resolver already, which fewer improvements we could expect from that. And we are also difficult to reduce possible timeout to maven central mirrors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [SPARK-42380][BUILD] Upgrade Apache Maven to 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1519098711
Long time no see @srowen , thanks for your review ~
I am not sure if the native http client will change the default timeout value to be same with wagon in the future. But I don't object to letting us wait and see if there will be any changes of the new version of Apache Maven :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bowenliang123 commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1503053830
Hi, after bumping to Maven 3.9.1 in Apache Kyuubi (https://github.com/apache/kyuubi/pull/4647), there are notable increasing failures in Maven builds when resolving Maven dependencies from the central mirror.
We are trying to investigate and improve the situation in https://github.com/apache/kyuubi/pull/4692, by increasing Maven Resolver Transport connection timeout back to original default connection timeout 60000ms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #40738: [BUILD] Test maven 3.9.1
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #40738:
URL: https://github.com/apache/spark/pull/40738#issuecomment-1504437932
<img width="1293" alt="image" src="https://user-images.githubusercontent.com/1475305/231329527-32cb53b8-6724-4461-8e26-5411ff81f000.png">
The last build took 1 hour and 7 meters, let me re-run GA more times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org