You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/19 03:12:23 UTC

[GitHub] [spark] LuciferYang opened a new pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

LuciferYang opened a new pull request #32227:
URL: https://github.com/apache/spark/pull/32227


   ### What changes were proposed in this pull request?
   There are 3 CVE problems were found after netty 4.1.51.Final as follows:
   
   - [CVE-2021-21409](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-21409)
   - [CVE-2021-21295](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-21295)
   - [CVE-2021-21290](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-21290)
   
   So the main change of this pr is upgrade netty-all to 4.1.61.Final avoid these potential risks.
   
   Another change is to clean up deprecated api usage: `Tiny caches have been merged into small caches and should use  PooledByteBufAllocator(boolean, int, int, int, int, int, int, boolean, int) api` to create `PooledByteBufAllocator`.
   
   ### Why are the changes needed?
   Upgrade netty-all to 4.1.61.Final avoid CVE problems.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Pass the Jenkins or GitHub Action
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822247974


   Can we fix the dependency list?
   
   ```
   To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
   diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3 b/dev/pr-deps/spark-deps-hadoop-2.7-hive-2.3
   index 646aa49..8e25ad7 100644
   --- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
   +++ b/dev/pr-deps/spark-deps-hadoop-2.7-hive-2.3
   @@ -191,7 +191,7 @@ metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
    metrics-json/4.1.1//metrics-json-4.1.1.jar
    metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
    minlog/1.3.0//minlog-1.3.0.jar
   -netty-all/4.1.51.Final//netty-all-4.1.63.Final.jar
   +netty-all/4.1.63.Final//netty-all-4.1.63.Final.jar
    objenesis/2.6//objenesis-2.6.jar
    okhttp/3.12.12//okhttp-3.12.12.jar
    okio/1.14.0//okio-1.14.0.jar
    ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615696604



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       ok 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822150881


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42122/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615532590



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       Although there is a description about this, could you add the link to Netty site about this change?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822391378


   **[Test build #137588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137588/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822455054


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42160/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822350867


   **[Test build #137612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137612/testReport)** for PR 32227 at commit [`13c2aea`](https://github.com/apache/spark/commit/13c2aeab5d2aacb83ca10b9cdaeb0180aecc5c87).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822445462


   **[Test build #137612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137612/testReport)** for PR 32227 at commit [`13c2aea`](https://github.com/apache/spark/commit/13c2aeab5d2aacb83ca10b9cdaeb0180aecc5c87).
    * This patch **fails PySpark pip packaging tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822252532


   I think there's a dependency resolution diff between Maven and SBT. You might have to manually exclude or override the version in `SparkBuild.scala` e.g.) `ExcludeDependencides` or `DependencyOverrides` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823724746


   thx @HyukjinKwon @dongjoon-hyun @srowen 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822446469


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137612/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822286234


   > I think there's a dependency resolution diff between Maven and SBT. You might have to manually exclude or override the version in SparkBuild.scala e.g.) ExcludeDependencides or DependencyOverrides
   
   https://github.com/apache/spark/pull/32230 try to solve this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822925552


   It seems that the fix of https://github.com/apache/spark/pull/32230# can solve the netty version conflict in the test
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822150881


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42122/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822193763


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822286234


   > I think there's a dependency resolution diff between Maven and SBT. You might have to manually exclude or override the version in SparkBuild.scala e.g.) ExcludeDependencides or DependencyOverrides
   
   https://github.com/apache/spark/pull/32230 (SPARK-35134)try to solve this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615551656



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       done ~




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822269399


   > I think there's a dependency resolution diff between Maven and SBT. You might have to manually exclude or override the version in SparkBuild.scala e.g.) ExcludeDependencides or DependencyOverrides
   
   Let me try this
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822192376


   **[Test build #137547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137547/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).
    * This patch **fails PySpark pip packaging tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822137508


   **[Test build #137547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137547/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang edited a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang edited a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822286234


   > I think there's a dependency resolution diff between Maven and SBT. You might have to manually exclude or override the version in SparkBuild.scala e.g.) ExcludeDependencides or DependencyOverrides
   
   https://github.com/apache/spark/pull/32230 (SPARK-35134) try to solve this problem first


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822257967


   **[Test build #137588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137588/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822446469


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137612/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823005681


   **[Test build #137671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137671/testReport)** for PR 32227 at commit [`c3bb7d6`](https://github.com/apache/spark/commit/c3bb7d603233059ad66a4dd1c165fc5f429407d3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823665244


   Merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615532590



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       Although there is a description about this, could you add the link to Netty site about this change in the PR description additionally?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823005681


   **[Test build #137671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137671/testReport)** for PR 32227 at commit [`c3bb7d6`](https://github.com/apache/spark/commit/c3bb7d603233059ad66a4dd1c165fc5f429407d3).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822411933


   **[Test build #137629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137629/testReport)** for PR 32227 at commit [`3e93efe`](https://github.com/apache/spark/commit/3e93efefd3c378e687ead53dbe1660a6fa51dfdd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822350867


   **[Test build #137612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137612/testReport)** for PR 32227 at commit [`13c2aea`](https://github.com/apache/spark/commit/13c2aeab5d2aacb83ca10b9cdaeb0180aecc5c87).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615628283



##########
File path: dev/deps/spark-deps-hadoop-2.7-hive-2.3
##########
@@ -191,7 +191,7 @@ metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
 metrics-json/4.1.1//metrics-json-4.1.1.jar
 metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
 minlog/1.3.0//minlog-1.3.0.jar
-netty-all/4.1.51.Final//netty-all-4.1.51.Final.jar
+netty-all/4.1.51.Final//netty-all-4.1.63.Final.jar

Review comment:
       `./dev/test-dependencies.sh --replace-manifest:` 
   thx @HyukjinKwon ~ learned this new skill ~
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615617804



##########
File path: dev/deps/spark-deps-hadoop-2.7-hive-2.3
##########
@@ -191,7 +191,7 @@ metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
 metrics-json/4.1.1//metrics-json-4.1.1.jar
 metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
 minlog/1.3.0//minlog-1.3.0.jar
-netty-all/4.1.51.Final//netty-all-4.1.51.Final.jar
+netty-all/4.1.51.Final//netty-all-4.1.63.Final.jar

Review comment:
       Oh, yeah, and seems like we should do `./dev/test-dependencies.sh --replace-manifest`:
   
   ```diff
   -netty-all/4.1.51.Final//netty-all-4.1.63.Final.jar
   +netty-all/4.1.63.Final//netty-all-4.1.63.Final.jar
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823137116


   @dongjoon-hyun @HyukjinKwon @srowen All test passed now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822193763


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137547/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822581780


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137629/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822150862






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822250549


   @HyukjinKwon ok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822581780


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137629/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823042664






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823066593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42199/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615696604



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       ok, but I think it will fail because of SPARK-35134




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822251336


   No, sorry @LuciferYang. Seems like the problemtic file was already fixed here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823114352


   **[Test build #137671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137671/testReport)** for PR 32227 at commit [`c3bb7d6`](https://github.com/apache/spark/commit/c3bb7d603233059ad66a4dd1c165fc5f429407d3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `class ImmutableBitSet(val numBits: Int, val bitsToSet: Int*) extends BitSet(numBits) `
     * `  case class CombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends TypeCoercionRule `
     * `case class GroupingExprRef(`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822137508


   **[Test build #137547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137547/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822409758


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137588/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615699327



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       There are some netty 4.1.50 dependencies in sbt test classpath, I'm trying to add some `ExcludeRule` to fix it now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615559177



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       Could you re-trigger your GitHub Action?
   - https://github.com/LuciferYang/spark/actions/runs/762178571




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822196938


   ![image](https://user-images.githubusercontent.com/1475305/115189179-ab4a6a80-a118-11eb-90c9-5bc31543b1a8.png)
   
   It seems that there are some old netty jars in `deps/jars` directory


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823116070


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137671/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822411933


   **[Test build #137629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137629/testReport)** for PR 32227 at commit [`3e93efe`](https://github.com/apache/spark/commit/3e93efefd3c378e687ead53dbe1660a6fa51dfdd).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822257967


   **[Test build #137588 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137588/testReport)** for PR 32227 at commit [`d280970`](https://github.com/apache/spark/commit/d280970694ab3e0e67e77e822cdcef391eb62bb4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823066593


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42199/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822340181


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42144/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822340181


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42144/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822247974


   Can we fix the dependency list?
   
   ```
   To update the manifest file, run './dev/test-dependencies.sh --replace-manifest'.
   diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3 b/dev/pr-deps/spark-deps-hadoop-2.7-hive-2.3
   index 646aa49..8e25ad7 100644
   --- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
   +++ b/dev/pr-deps/spark-deps-hadoop-2.7-hive-2.3
   @@ -191,7 +191,7 @@ metrics-jmx/4.1.1//metrics-jmx-4.1.1.jar
    metrics-json/4.1.1//metrics-json-4.1.1.jar
    metrics-jvm/4.1.1//metrics-jvm-4.1.1.jar
    minlog/1.3.0//minlog-1.3.0.jar
   -netty-all/4.1.51.Final//netty-all-4.1.63.Final.jar
   +netty-all/4.1.63.Final//netty-all-4.1.63.Final.jar
    objenesis/2.6//objenesis-2.6.jar
    okhttp/3.12.12//okhttp-3.12.12.jar
    okio/1.14.0//okio-1.14.0.jar
    ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-823116070


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137671/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822406272


   cherry-pick 3e93efe from SPARK-35134 to check GA test result. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822409758


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137588/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822455008


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42160/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822451375


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42160/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822559035


   **[Test build #137629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137629/testReport)** for PR 32227 at commit [`3e93efe`](https://github.com/apache/spark/commit/3e93efefd3c378e687ead53dbe1660a6fa51dfdd).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822251525


   It's weird that it fails ..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32227:
URL: https://github.com/apache/spark/pull/32227#discussion_r615558641



##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java
##########
@@ -162,7 +162,6 @@ public static PooledByteBufAllocator createPooledByteBufAllocator(
       Math.min(PooledByteBufAllocator.defaultNumDirectArena(), allowDirectBufs ? numCores : 0),
       PooledByteBufAllocator.defaultPageSize(),
       PooledByteBufAllocator.defaultMaxOrder(),
-      allowCache ? PooledByteBufAllocator.defaultTinyCacheSize() : 0,

Review comment:
       Thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen closed pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
srowen closed pull request #32227:
URL: https://github.com/apache/spark/pull/32227


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822455054


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42160/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822318147






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] LuciferYang commented on pull request #32227: [WIP][SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #32227:
URL: https://github.com/apache/spark/pull/32227#issuecomment-822223651


   I find that there is always netty 4.1.50 in classpath now, even for the master branch , this will cause some tests fail, and I created them https://issues.apache.org/jira/browse/SPARK-35134 to track this problem.
   
   I try to execute  
   
   ```
   mvn dependency:tree -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
   ```
    there was no netty 4.1.50 dependency in the results.
   
   Do you have any good suggestions? @dongjoon-hyun @HyukjinKwon @srowen 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org