You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/24 22:50:10 UTC

[GitHub] [spark] HeartSaVioR opened a new pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

HeartSaVioR opened a new pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687
 
 
   ### What changes were proposed in this pull request?
   
   This patch changes the tool tip string in Structured Streaming UI graphs to show batch ID (and timestamp as well) instead of only showing timestamp, which was a key for DStream but no longer a key for Structured Streaming.
   
   This patch does some refactoring as there're some spots on confusion between js file for streaming and structured streaming.
   
   ### Why are the changes needed?
   
   In Structured Streaming, everything is aligned for "batch ID" where the UI is only showing timestamp - end users have to manually find and correlate batch ID and the timestamp which is clearly a huge pain.
   
   ### Does this PR introduce any user-facing change?
   
   No
   
   ### How was this patch tested?
   
   Manually tested. Screenshots:
   
   ![Screen Shot 2020-02-25 at 7 22 38 AM](https://user-images.githubusercontent.com/1317309/75197701-40b2ce80-57a2-11ea-9578-c2eb2d1091de.png)
   ![Screen Shot 2020-02-25 at 7 22 44 AM](https://user-images.githubusercontent.com/1317309/75197704-427c9200-57a2-11ea-9439-e0a8303d0860.png)
   ![Screen Shot 2020-02-25 at 7 22 58 AM](https://user-images.githubusercontent.com/1317309/75197706-43152880-57a2-11ea-9617-1276c3ba181e.png)
   ![Screen Shot 2020-02-25 at 7 23 04 AM](https://user-images.githubusercontent.com/1317309/75197708-43152880-57a2-11ea-9de2-7d37eaf88102.png)
   ![Screen Shot 2020-02-25 at 7 23 31 AM](https://user-images.githubusercontent.com/1317309/75197710-43adbf00-57a2-11ea-9ae4-4e292de39c36.png)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383562674
 
 

 ##########
 File path: core/src/main/resources/org/apache/spark/ui/static/structured-streaming-page.js
 ##########
 @@ -106,12 +106,12 @@ function drawAreaStack(id, labels, values, minX, maxX, minY, maxY) {
         .on('mouseover', function(d) {
             var tip = '';
             var idx = 0;
-            var _values = timeToValues[d._x]
+            var _values = formattedTimeToValues[d._x]
 
 Review comment:
   I've also renamed to add `formatted`, as it requires formatted time string for x axis of input.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590652356
 
 
   **[Test build #118889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118889/testReport)** for PR 27687 at commit [`5c2534a`](https://github.com/apache/spark/commit/5c2534a1884a2485af8e6f74d279c1ef6a7a6b50).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591383
 
 
   **[Test build #118886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118886/testReport)** for PR 27687 at commit [`69eaa40`](https://github.com/apache/spark/commit/69eaa40fd6e86e6a2c52e7523eb6bda2ad490bc2).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590590918
 
 
   cc. @tdas @zsxwing @gaborgsomogyi @uncleGen @xuanyuanking 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590638123
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590637599
 
 
   **[Test build #118886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118886/testReport)** for PR 27687 at commit [`69eaa40`](https://github.com/apache/spark/commit/69eaa40fd6e86e6a2c52e7523eb6bda2ad490bc2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590650773
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383562219
 
 

 ##########
 File path: core/src/main/resources/org/apache/spark/ui/static/streaming-page.js
 ##########
 @@ -171,7 +171,7 @@ function drawTimeline(id, data, minX, maxX, minY, maxY, unitY, batchInterval) {
             .attr("cy", function(d) { return y(d.y); })
             .attr("r", function(d) { return isFailedBatch(d.x) ? "2" : "3";})
             .on('mouseover', function(d) {
-                var tip = formatYValue(d.y) + " " + unitY + " at " + timeFormat[d.x];
+                var tip = formatYValue(d.y) + " " + unitY + " at " + timeTipStrings[d.x];
 
 Review comment:
   So we explicitly use different variable for showing tip as extension point - input from DStream and input and Structured Streaming will be different.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590638127
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118886/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590620270
 
 
   LGTM except for some minor issues.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591383
 
 
   **[Test build #118886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118886/testReport)** for PR 27687 at commit [`69eaa40`](https://github.com/apache/spark/commit/69eaa40fd6e86e6a2c52e7523eb6bda2ad490bc2).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590650781
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23638/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590638127
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118886/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591801
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-591140854
 
 
   Thanks all for reviewing and merging!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590650746
 
 
   Thanks @sarutak for the review! I've addressed review comments. Please take a look again. Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590696491
 
 
   **[Test build #118889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118889/testReport)** for PR 27687 at commit [`5c2534a`](https://github.com/apache/spark/commit/5c2534a1884a2485af8e6f74d279c1ef6a7a6b50).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590638123
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590697103
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118889/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590650773
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590697097
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590650781
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23638/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591810
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23635/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591801
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590697097
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383562823
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
 ##########
 @@ -70,11 +70,30 @@ private[ui] class StreamingQueryStatisticsPage(parent: StreamingQueryTab)
     <script>{Unparsed(js)}</script>
   }
 
-  def generateVar(values: Array[(Long, ju.Map[String, JLong])]): Seq[Node] = {
+  def generateTimeTipStrings(values: Array[(Long, Long)]): Seq[Node] = {
 
 Review comment:
   This is for the input of `streaming js`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383595956
 
 

 ##########
 File path: core/src/main/resources/org/apache/spark/ui/static/structured-streaming-page.js
 ##########
 @@ -106,12 +106,12 @@ function drawAreaStack(id, labels, values, minX, maxX, minY, maxY) {
         .on('mouseover', function(d) {
             var tip = '';
             var idx = 0;
-            var _values = timeToValues[d._x]
+            var _values = formattedTimeToValues[d._x]
 
 Review comment:
   nit: You have a chance to append ";" now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590697103
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118889/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590652356
 
 
   **[Test build #118889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118889/testReport)** for PR 27687 at commit [`5c2534a`](https://github.com/apache/spark/commit/5c2534a1884a2485af8e6f74d279c1ef6a7a6b50).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383562905
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
 ##########
 @@ -70,11 +70,30 @@ private[ui] class StreamingQueryStatisticsPage(parent: StreamingQueryTab)
     <script>{Unparsed(js)}</script>
   }
 
-  def generateVar(values: Array[(Long, ju.Map[String, JLong])]): Seq[Node] = {
+  def generateTimeTipStrings(values: Array[(Long, Long)]): Seq[Node] = {
+    val js = "var timeTipStrings = {};\n" + values.map { case (batchId, time) =>
+      val formattedTime = SparkUIUtils.formatBatchTime(time, 1, showYYYYMMSS = false)
+      s"timeTipStrings[$time] = 'batch $batchId ($formattedTime)';"
+    }.mkString("\n")
+
+    <script>{Unparsed(js)}</script>
+  }
+
+  def generateFormattedTimeTipStrings(values: Array[(Long, Long)]): Seq[Node] = {
 
 Review comment:
   This is for the input of `structured streaming js`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak edited a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590686230
 
 
   LGTM. I'll wait for comment/approval from other developers.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] asfgit closed pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590591810
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23635/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-590686230
 
 
   LGTM. I'll wait for comment/approvement from other developers.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383594619
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala
 ##########
 @@ -114,6 +133,9 @@ private[ui] class StreamingQueryStatisticsPage(parent: StreamingQueryTab)
   def generateStatTable(query: StreamingQueryUIData): Seq[Node] = {
     val batchTimes = withNoProgress(query,
       query.recentProgress.map(p => df.parse(p.timestamp).getTime), Array.empty[Long])
+    val batchToTimestamps = withNoProgress(query,
 
 Review comment:
   Can we create `batchTimes` like `batchToTimestamps.map(_._2)` to avoid having duplicate code?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zsxwing commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
zsxwing commented on issue #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#issuecomment-591134718
 
 
   LGTM. Merging to master and 3.0.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs

Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #27687: [SPARK-30943][SS] Show "batch ID" in tool tip string for Structured Streaming UI graphs
URL: https://github.com/apache/spark/pull/27687#discussion_r383589380
 
 

 ##########
 File path: streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala
 ##########
 @@ -140,6 +140,16 @@ private[ui] class StreamingPage(parent: StreamingTab)
     <script>{Unparsed(js)}</script>
   }
 
+  private def generateTimeTipStrings(times: Seq[Long]): Seq[Node] = {
+    val js = "var timeTipStrings = {};\n" + times.map { time =>
+      val formattedTime =
+        SparkUIUtils.formatBatchTime(time, listener.batchDuration, showYYYYMMSS = false)
+      s"timeTipStrings[$time] = '$formattedTime';"
+    }.mkString("\n")
 
 Review comment:
   This logic seems to be the same as [here](https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L133).
   Can we do like `s"timeTipStrings[$time] = timeFormat[$time];` or pull out the common code?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org