You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/06 04:38:58 UTC

[GitHub] [spark] HeartSaVioR opened a new pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

HeartSaVioR opened a new pull request #31056:
URL: https://github.com/apache/spark/pull/31056


   ### What changes were proposed in this pull request?
   
   This PR proposes to adjust the order of check in KafkaTokenUtil.needTokenUpdate, so that short-circuit applies on the non-delegation token cases (insecure + secured without delegation token) and remedies the performance regression heavily.
   
   ### Why are the changes needed?
   
   There's a serious performance regression between Spark 2.4 vs Spark 3.0 on read path against Kafka data source.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Manually ran a reproducer (https://github.com/codegorillauk/spark-kafka-read with modification to just count instead of writing to Kafka topic) with measuring the time.
   
   > the branch applying the change with adding measurement
   
   https://github.com/HeartSaVioR/spark/commits/debug-SPARK-33635-v3.0.1
   
   > the branch only adding measurement
   
   https://github.com/HeartSaVioR/spark/commits/debug-original-ver-SPARK-33635-v3.0.1
   
   > the result (before the fix)
   
   count: 10280000
   Took 41.634007047 secs
   
   21/01/06 13:16:07 INFO KafkaDataConsumer: debug ver. 17-original
   21/01/06 13:16:07 INFO KafkaDataConsumer: Total time taken to retrieve: 82118 ms
   
   > the result (after the fix)
   
   count: 10280000
   Took 7.964058475 secs
   
   21/01/06 13:08:22 INFO KafkaDataConsumer: debug ver. 17
   21/01/06 13:08:22 INFO KafkaDataConsumer: Total time taken to retrieve: 987 ms


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755074448


   I just fixed the smallest thing as a short-term solution (the problem will persist for end users using Kafka delegation token), as Spark 3.1.0 RC vote is happening and I don't want to drag the release too much.
   
   It would be ideal if we deal with long-term solution (making overhead of `HadoopDelegationTokenManager.isServiceEnabled` be minor, or reduce occurrence of the check heavily) in time, but just wanted to be safe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31056:
URL: https://github.com/apache/spark/pull/31056


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755086157


   **[Test build #133711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133711/testReport)** for PR 31056 at commit [`f5decdd`](https://github.com/apache/spark/commit/f5decddd5bec82e3902a40a1226ab46e0fc7dabe).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755126291


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38299/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755077964


   The further optimization should be depending on the possibility of "changes" in SparkConf on the fly. If we don't allow it (or at least Kafka/token related configuration), we just need to call `HadoopDelegationTokenManager.isServiceEnabled` once per JVM, and cache it to avoid calling per each get.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755102360


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133711/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755126291


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38299/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755100551


   cc @HyukjinKwon since he is the release manager of Spark 3.1.0.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755082088


   In addition, the time in `Total time taken to retrieve` checks pure overhead compared to Spark 2.4, but I ran the application with my local dev where parallelism is not fully enabled and I also have to do some other thing in parallel, so the value should be pretty much smaller (probably trivial) if the test environment is isolated properly with enough power to enable full parallelism.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755098940


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38299/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755110492


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38299/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755074448


   I just made the smallest change as a short-term solution (the problem will persist for end users using Kafka delegation token), as Spark 3.1.0 RC vote is happening and I don't want to drag the release too much.
   
   It would be ideal if we deal with long-term solution (making overhead of `HadoopDelegationTokenManager.isServiceEnabled` be minor, or reduce occurrence of the check heavily) in time, but just wanted to be safe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755082088


   In addition, the time in `Total time taken to retrieve` checks pure overhead compared to Spark 2.4, but I ran the application with my local dev where parallelism is not fully enabled (just ran with `local[3]`) and I also have to do some other thing in parallel, so the value should be pretty much smaller (probably trivial) if the test environment is isolated properly with enough power to enable full parallelism.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gaborgsomogyi commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755222402


   Since `SparkConf` is static we can cache the first time calculated values. I've started to work on the long term solution.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755102360


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133711/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755080748


   cc. @tdas @zsxwing @jose-torres @viirya @gaborgsomogyi @xuanyuanking 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755081307


   **[Test build #133711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133711/testReport)** for PR 31056 at commit [`f5decdd`](https://github.com/apache/spark/commit/f5decddd5bec82e3902a40a1226ab46e0fc7dabe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31056: [SPARK-33635][SS] Adjust the order of check in KafkaTokenUtil.needTokenUpdate to remedy perf regression

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31056:
URL: https://github.com/apache/spark/pull/31056#issuecomment-755081307


   **[Test build #133711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133711/testReport)** for PR 31056 at commit [`f5decdd`](https://github.com/apache/spark/commit/f5decddd5bec82e3902a40a1226ab46e0fc7dabe).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org