You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2023/07/12 14:06:34 UTC
[GitHub] [arrow] raulcd opened a new pull request, #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
raulcd opened a new pull request, #36640:
URL: https://github.com/apache/arrow/pull/36640
### Rationale for this change
It does seem that the only way to shadow Netty version is to modify the Pom for previous versions.
### What changes are included in this PR?
Try to patch version of Netty on the pom when cloning Spark.
### Are these changes tested?
Archery integration tests
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd closed pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
URL: https://github.com/apache/arrow/pull/36640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "kiszk (via GitHub)" <gi...@apache.org>.
kiszk commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1635958216
cc @BryanCutler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1656010264
I am closing this PR as upgrading to netty 4.1.96 fixed the issue reverting the regression introduced on 4.1.94.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "kiszk (via GitHub)" <gi...@apache.org>.
kiszk commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1636002003
@raulcd Sorry for confusing you. I am fine with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1632646249
This is not going to be enough because from my understanding Spark also pins the versions used on Hadoop here:
https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203
and I am not sure we want to patch those too.
@kiszk any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a diff in pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "kiszk (via GitHub)" <gi...@apache.org>.
kiszk commented on code in PR #36640:
URL: https://github.com/apache/arrow/pull/36640#discussion_r1268256015
##########
ci/scripts/integration_spark.sh:
##########
@@ -45,6 +45,13 @@ export MAVEN_OPTS="${MAVEN_OPTS} -Dorg.slf4j.simpleLogger.log.org.apache.maven.c
pushd ${spark_dir}
+ # Due to CVE-2023-34462 we upgraded to a memory netty version which is incompatible
+ # with previous spark versions. Patch the pom to use newer version.
+ sed -i.bak -E -e \
Review Comment:
Hmm, [this line](https://github.com/ursacomputing/crossbow/actions/runs/5532636163/jobs/10095003367#step:6:13303) may also say to use `4.1.93.final`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on a diff in pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #36640:
URL: https://github.com/apache/arrow/pull/36640#discussion_r1268309957
##########
ci/scripts/integration_spark.sh:
##########
@@ -45,6 +45,13 @@ export MAVEN_OPTS="${MAVEN_OPTS} -Dorg.slf4j.simpleLogger.log.org.apache.maven.c
pushd ${spark_dir}
+ # Due to CVE-2023-34462 we upgraded to a memory netty version which is incompatible
+ # with previous spark versions. Patch the pom to use newer version.
+ sed -i.bak -E -e \
Review Comment:
Thanks, I'll investigate, I might have messed up on my tests :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1635998921
@kiszk if I understand correctly you are suggesting to also patch the versions here: https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203 to see if the job changing the versions succeeds (at least on master spark)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LuciferYang commented on a diff in pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #36640:
URL: https://github.com/apache/arrow/pull/36640#discussion_r1266614969
##########
ci/scripts/integration_spark.sh:
##########
@@ -45,6 +45,13 @@ export MAVEN_OPTS="${MAVEN_OPTS} -Dorg.slf4j.simpleLogger.log.org.apache.maven.c
pushd ${spark_dir}
+ # Due to CVE-2023-34462 we upgraded to a memory netty version which is incompatible
+ # with previous spark versions. Patch the pom to use newer version.
+ sed -i.bak -E -e \
Review Comment:
I run this `sed` statement locally, but I found that the `netty.version` in the `pom.xml` still `4.1.93.Final `
Maybe we can test `build/mvn versions:set-property -Dproperty=netty.version -DnewVersion=4.1.94.Final -DgenerateBackupPoms=false`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "kiszk (via GitHub)" <gi...@apache.org>.
kiszk commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1635956706
LGTM.
It looks like a practical solution to resolve a chicken-and-egg problem.
We need to ensure the latest Arrow works well with Spark. Then, we can release the latest Arrow with Netty 4.1.94. After that, Spark would update [the version of Arrow](https://github.com/apache/spark/blob/master/pom.xml#L219-L228).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1632596719
@github-actions crossbow submit *spark*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LuciferYang commented on a diff in pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #36640:
URL: https://github.com/apache/arrow/pull/36640#discussion_r1266614969
##########
ci/scripts/integration_spark.sh:
##########
@@ -45,6 +45,13 @@ export MAVEN_OPTS="${MAVEN_OPTS} -Dorg.slf4j.simpleLogger.log.org.apache.maven.c
pushd ${spark_dir}
+ # Due to CVE-2023-34462 we upgraded to a memory netty version which is incompatible
+ # with previous spark versions. Patch the pom to use newer version.
+ sed -i.bak -E -e \
Review Comment:
I run this `sed` statement locally, but I found that the `netty.version` in the `pom.xml` still `4.1.93.Final `
Maybe we can test `build/mvn versions:set-property -Dproperty=netty.version -DnewVersion=4.1.94.Final -DgenerateBackupPoms=false`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "kiszk (via GitHub)" <gi...@apache.org>.
kiszk commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1635941621
This PR is motivated by https://github.com/apache/arrow/issues/36332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] LuciferYang commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1639308187
> @kiszk if I understand correctly you are suggesting to also patch the versions here: https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L186-L203 to see if the job changing the versions succeeds (at least on master spark)?
This just used by dependencies change check(like a golden file), will not affect the actual netty version usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1636003875
but the spark jobs still fail unless we also patch those. I mean the current change doesn't seem enough to make our CI integration jobs with spark successful as seen on the crossbow report comment here: https://github.com/apache/arrow/pull/36640#issuecomment-1632601383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1632601383
Revision: 44ec8f85ae20b3e7ed08b0fbd0080be3a5987b61
Submitted crossbow builds: [ursacomputing/crossbow @ actions-e71df5add3](https://github.com/ursacomputing/crossbow/branches/all?query=actions-e71df5add3)
|Task|Status|
|----|------|
|test-conda-python-3.10-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/5532636163/jobs/10095003367)|
|test-conda-python-3.8-spark-v3.1.2|[](https://github.com/ursacomputing/crossbow/actions/runs/5532635517/jobs/10095001944)|
|test-conda-python-3.9-spark-v3.2.0|[](https://github.com/ursacomputing/crossbow/actions/runs/5532635832/jobs/10095002531)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #36640: GH-36332: [CI][Java] Patch spark to use Netty 4.1.94.Final on our integration tests
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36640:
URL: https://github.com/apache/arrow/pull/36640#issuecomment-1632597201
:warning: GitHub issue #36332 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org