You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/22 01:34:06 UTC

[GitHub] [spark] baohe-zhang opened a new pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

baohe-zhang opened a new pull request #30126:
URL: https://github.com/apache/spark/pull/30126


   ### What changes were proposed in this pull request?
   This patch separates the view permission checks from the getAppUi in FsHistoryServerProvider, thus enabling SHS to do view permissions check of a given attempt for a given user without rebuilding the UI. This is achieved by adding a method "checkUIViewPermissions(appId: String, attemptId: Option[String], user: String): Boolean" to many layers of history server components. Currently, this feature is useful for event log download.
   
   
   ### Why are the changes needed?
   Right now, when we want to download the event logs from the spark history server, SHS will need to parse entire the event log to rebuild UI, and this is just for view permission checks. UI rebuilding is a time-consuming and memory-intensive task, especially for large logs. However, this process is unnecessary for event log download. With this patch, UI rebuild can be skipped when downloading event logs from the history server. Thus the time of downloading a GB scale event log can be reduced from several minutes to several seconds, and the memory consumption of UI rebuilding can be avoided.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   Added a unit test for fsHistoryProvider.checkUIViewPermissions(). Also did some manual tests to verify the download speed can be drastically improved and the view permissions check works properly.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716961983


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34911/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] baohe-zhang commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716994448


   Thanks for the review!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716992502


   Thanks for your contribution! Merged into master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716991805






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714168075


   **[Test build #130117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130117/testReport)** for PR 30126 at commit [`a5538f9`](https://github.com/apache/spark/commit/a5538f954eb1303db3c2d9ec271b3f325d87440b).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714168100


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130117/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714193726


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34726/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714218932


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130118/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715102743


   cc. @gengliangwang @vanzin @squito 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716962002






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714193718






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715461860






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714168095


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714173134


   **[Test build #130118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130118/testReport)** for PR 30126 at commit [`990ab15`](https://github.com/apache/spark/commit/990ab152df7407df1da1ea93680ece48005a8651).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716941185


   **[Test build #130309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130309/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716941185


   **[Test build #130309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130309/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716940154






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715425748


   **[Test build #130209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130209/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715448000


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34809/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716955545


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34911/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714191435






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714168095






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714191438






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] mridulm commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715881096


   +CC @zoelin7


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715461860






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715511491


   **[Test build #130209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130209/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] mridulm commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715882396


   Also, CC @zhouyejoe 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716962002






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714191422


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34725/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716991805






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715461846


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34809/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] baohe-zhang commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714570041


   Hi @HeartSaVioR , could you review this pr when you get a chance?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-716990746


   **[Test build #130309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130309/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] baohe-zhang commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
baohe-zhang commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715426513


   @HeartSaVioR I added the test case and verified it can pass with this patch and it would fail without this patch.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714184103


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34725/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715512484






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714165779


   **[Test build #130117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130117/testReport)** for PR 30126 at commit [`a5538f9`](https://github.com/apache/spark/commit/a5538f954eb1303db3c2d9ec271b3f325d87440b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714185293


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34726/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715425748


   **[Test build #130209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130209/testReport)** for PR 30126 at commit [`d8d78c7`](https://github.com/apache/spark/commit/d8d78c7cb0eb757231edeaf5d27fd59bf77c02e3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714218065


   **[Test build #130118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130118/testReport)** for PR 30126 at commit [`990ab15`](https://github.com/apache/spark/commit/990ab152df7407df1da1ea93680ece48005a8651).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR closed pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #30126:
URL: https://github.com/apache/spark/pull/30126


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714193706


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34726/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715512484






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714173134


   **[Test build #130118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130118/testReport)** for PR 30126 at commit [`990ab15`](https://github.com/apache/spark/commit/990ab152df7407df1da1ea93680ece48005a8651).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714165779


   **[Test build #130117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130117/testReport)** for PR 30126 at commit [`a5538f9`](https://github.com/apache/spark/commit/a5538f954eb1303db3c2d9ec271b3f325d87440b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714218927






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-715101308


   Looks good overall. Could we have a test to confirm event log download doesn't trigger ui load? You can add below code to HistoryServerSuite (probably after the test "ui and api authorization checks").
   
   ```
     test("SPARK-33215: speed up event log download by skipping UI rebuild") {
       val appId = "local-1430917381535"
   
       stop()
       init()
   
       val port = server.boundPort
       val testUrls = Seq(
         s"http://localhost:$port/api/v1/applications/$appId/logs",
         s"http://localhost:$port/api/v1/applications/$appId/1/logs",
         s"http://localhost:$port/api/v1/applications/$appId/2/logs")
   
       testUrls.foreach { url =>
         TestUtils.httpResponseCode(new URL(url))
       }
       assert(server.cacheMetrics.loadCount.getCount === 0, "downloading event log shouldn't load ui")
     }
   ```
   
   This would fail without the patch and pass with the patch.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30126: [SPARK-33215][WEBUI] Speed up event log download by skipping UI rebuild

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30126:
URL: https://github.com/apache/spark/pull/30126#issuecomment-714218927


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org