You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/04 01:05:14 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
HeartSaVioR opened a new pull request #31464:
URL: https://github.com/apache/spark/pull/31464
### What changes were proposed in this pull request?
This PR proposes to expose the number of truncated paths in Utils.buildLocationMetadata(), with relaxing space usage a bit (around 10 chars).
Suppose the first 2 of 5 paths are only fit to the threshold, the outputs between the twos are below:
* before the change: [path1, path2]
* after the change: [path1, path2, ... 3 more]
### Why are the changes needed?
SPARK-31793 silently truncates the paths hence end users can't indicate how many paths are truncated, and even more, whether paths are truncated or not.
### Does this PR introduce _any_ user-facing change?
Yes, the location metadata will also show how many paths are truncated (not shown), instead of silently truncated.
### How was this patch tested?
Modified UTs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773079984
**[Test build #134853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134853/testReport)** for PR 31464 at commit [`56abfe2`](https://github.com/apache/spark/commit/56abfe21b6216eb3b58e96b13c3974a52cbb10e2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773072398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773031370
**[Test build #134853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134853/testReport)** for PR 31464 at commit [`56abfe2`](https://github.com/apache/spark/commit/56abfe21b6216eb3b58e96b13c3974a52cbb10e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773031370
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773031370
**[Test build #134853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134853/testReport)** for PR 31464 at commit [`56abfe2`](https://github.com/apache/spark/commit/56abfe21b6216eb3b58e96b13c3974a52cbb10e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773137525
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773072398
Hmm... I'll need to fix the test as a lot of tests are relying on the previous format (`Location: <className>[]`). I'd like to hear the voice on the new output format before fixing these.
(If tests are simply checking it via regex then moving `(N paths)` to the first part of `[]` would work, but it seems clearer to have it separately.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773010201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134849/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773155274
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773046503
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773072398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772988761
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39436/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773053389
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773222384
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134861/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569907071
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
I'm okay either way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773081377
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773155309
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772990065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39436/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31464:
URL: https://github.com/apache/spark/pull/31464
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569904399
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
Oh sorry, I meant truncated by `StringUtils.abbreviate`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569906399
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
Another idea (simpler) is that just adding the number of paths in prior, and leave it as it is. The tail part could be abbreviated, but we still show the number of paths in total. (It'll be no longer `... N more`. It's like `Total N paths: `.) Is alternative idea OK for everyone?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773048207
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569906399
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
Another idea (simpler) is that just adding the number of paths in prior, and leave it as it is. The tail part could be abbreviated, but we still show the number of paths in total. (It'll be no longer `... N more`. It's like `(total N paths)[path1, path2]`.) Is alternative idea OK for everyone?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773010201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134849/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772951810
**[Test build #134849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134849/testReport)** for PR 31464 at commit [`f0a02fd`](https://github.com/apache/spark/commit/f0a02fd71125a30c310a7fe89b70936a7a9ba95c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773112085
**[Test build #134861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134861/testReport)** for PR 31464 at commit [`e766209`](https://github.com/apache/spark/commit/e766209400f7a1767d48d02e3b9e142582ac3dba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772990065
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39436/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772957795
Yeah, I think this is better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772951810
**[Test build #134849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134849/testReport)** for PR 31464 at commit [`f0a02fd`](https://github.com/apache/spark/commit/f0a02fd71125a30c310a7fe89b70936a7a9ba95c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773667721
@HyukjinKwon @cloud-fan @gengliangwang Appreciate another round of review. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773104094
I was wrong about the amount of broken UTs. They were just 2 and I just fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773222384
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134861/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773072398
Hmm... I'll need to fix the test as a lot of tests are relying on the previous format (`Location: <className>[]`). I'd like to hear the voice on the new output format before fixing these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569905586
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
I think making the actual metadata data prettier is fine even though it's abbreviate in printed out result.
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
I think making the actual metadata data prettier is fine even though it's abbreviated in the printed out result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773198283
**[Test build #134861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134861/testReport)** for PR 31464 at commit [`e766209`](https://github.com/apache/spark/commit/e766209400f7a1767d48d02e3b9e142582ac3dba).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773698884
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772970020
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39436/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569900783
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
`... number more` matches our redact pattern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773155309
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773053389
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569906864
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
The original PR reduced the size of memory usage, which is fine. I think the current change is okay as well to make sure raw metadata shows that the paths are abbreviate and truncated.
We will just have to make sure it looks pretty in both printed output and raw metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773741742
Thanks all for reviewing and merging!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773081377
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569936135
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
I've reflected the second option. It's probably less beautiful but easily avoid possible glitches. I've also updated the PR title/description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773072398
Hmm... I'll need to fix the test as a lot of tests are relying on the previous format (`Location: <className> []`). I'd like to hear the voice on the new output format before fixing these.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773698884
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773007828
**[Test build #134849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134849/testReport)** for PR 31464 at commit [`f0a02fd`](https://github.com/apache/spark/commit/f0a02fd71125a30c310a7fe89b70936a7a9ba95c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569908676
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
I'm okay either way, too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773112085
**[Test build #134861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134861/testReport)** for PR 31464 at commit [`e766209`](https://github.com/apache/spark/commit/e766209400f7a1767d48d02e3b9e142582ac3dba).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31464:
URL: https://github.com/apache/spark/pull/31464
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569904841
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
Ah OK. That looks to be a tricky one now...
Would we like to apply strict length limit on the metadata location string, so that StringUtils.abbreviate won't touch the abbreviated form of output? It's a bit odd we try to abbreviate but apply soft limit & add some information to the abbreviated form which leads to exceed the limit, and abbreviate again to break the abbreviated form.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773056239
Appreciate another round of review based on the changed output. I'll try out alternative if the output doesn't look good for us. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773081377
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773081377
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #31464:
URL: https://github.com/apache/spark/pull/31464#discussion_r569894341
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2989,6 +2989,9 @@ private[spark] object Utils extends Logging {
metadata.append(paths(index).toString)
index += 1
}
+ if (paths.length > index) {
+ metadata.append(s", ... ${paths.length - index} more")
Review comment:
This appended info is probably redacted in https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L62 and it won't be shown in plan output.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of total paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-773079984
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772946664
cc.ing @gengliangwang @cloud-fan @HyukjinKwon @maropu who are author/reviewers of #28610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31464: [SPARK-34339][CORE][SQL] Expose the number of truncated paths in Utils.buildLocationMetadata()
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31464:
URL: https://github.com/apache/spark/pull/31464#issuecomment-772945967
While I've marked SPARK-34339 as improvement, I also feel this may be considered as a bug, as the new output brought by SPARK-31793 brings confusion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org