You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/13 20:51:01 UTC
[GitHub] [spark] erenavsarogullari opened a new pull request #28208:
[SPARK-31440][SQL] Improve SQL Rest API
erenavsarogullari opened a new pull request #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208
### What changes were proposed in this pull request?
SQL Rest API exposes query execution metrics as Public API. This PR aims to apply following improvements on SQL Rest API by aligning Spark-UI.
**Proposed Improvements:**
1- Support Physical Operations and group metrics per operation by aligning Spark UI.
2- `nodeId` can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics.
3- Filter `blank` metrics by aligning with Spark UI - SQL Tab
4- Remove `\n` from `metricValue`
5- `planDescription` can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans).
6- `metrics` attribute needs to be exposed at the bottom order as `metricDetails`. This order matches with Spark UI by highlighting with execution order.
### Why are the changes needed?
Proposed improvements provides more useful (e.g: physical operations and metrics correlation, grouping) and clear (e.g: filtering blank metrics, removing line breakers) result for the end-user.
### Does this PR introduce any user-facing change?
Yes. Please find both current and improved versions of the results as attached for following SQL Rest Endpoint:
```
curl -X GET http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true
```
**Current version:**
https://issues.apache.org/jira/secure/attachment/12999821/current_version.json
**Improved version:**
https://issues.apache.org/jira/secure/attachment/12999822/improved_version.json
### How was this patch tested?
1. New Unit tests are added.
2. Also, patch has been tested manually through both **Spark Core** and **History Server** Rest APIs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613183714
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613092827
**[Test build #121221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121221/testReport)** for PR 28208 at commit [`9754b66`](https://github.com/apache/spark/commit/9754b66ca26df7d1d90d7b6d6ef8f9fcbad2599a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102116
**[Test build #121223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121223/testReport)** for PR 28208 at commit [`a26cbbe`](https://github.com/apache/spark/commit/a26cbbec135cfd9ea7f8a4abed9a3bd8666f604f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093331
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102709
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093607
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613183722
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093336
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093336
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613183722
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121223/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613183714
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093607
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093591
**[Test build #121221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121221/testReport)** for PR 28208 at commit [`9754b66`](https://github.com/apache/spark/commit/9754b66ca26df7d1d90d7b6d6ef8f9fcbad2599a).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102709
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093331
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121221/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102116
**[Test build #121223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121223/testReport)** for PR 28208 at commit [`a26cbbe`](https://github.com/apache/spark/commit/a26cbbec135cfd9ea7f8a4abed9a3bd8666f604f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102715
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25911/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613183010
**[Test build #121223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121223/testReport)** for PR 28208 at commit [`a26cbbe`](https://github.com/apache/spark/commit/a26cbbec135cfd9ea7f8a4abed9a3bd8666f604f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28208: [SPARK-31440][SQL]
Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613093617
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121221/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613102715
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25911/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-613092827
**[Test build #121221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121221/testReport)** for PR 28208 at commit [`9754b66`](https://github.com/apache/spark/commit/9754b66ca26df7d1d90d7b6d6ef8f9fcbad2599a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erenavsarogullari commented on issue #28208:
[SPARK-31440][SQL] Improve SQL Rest API
Posted by GitBox <gi...@apache.org>.
erenavsarogullari commented on issue #28208: [SPARK-31440][SQL] Improve SQL Rest API
URL: https://github.com/apache/spark/pull/28208#issuecomment-614245275
cc @gengliangwang @vanzin @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org