You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/04 09:31:30 UTC
[GitHub] [spark] ulysses-you opened a new pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
ulysses-you opened a new pull request #31471:
URL: https://github.com/apache/spark/pull/31471
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add some info log around commit log.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Th commit job is a heavy option and we have seen many times Spark block at this code place due to the slow rpc with namenode or other.
It's better to record the time that commit job cost.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, more info log.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Not need.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774942341
**[Test build #134999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134999/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773773238
**[Test build #134906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134906/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773187923
**[Test build #134868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134868/testReport)** for PR 31471 at commit [`def94f1`](https://github.com/apache/spark/commit/def94f16f0704c98c69c2cbd62e4ac3229619180).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773255967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959
Actually I have been thinking about this - while I think this helps to track down the elapsed time on committing job, there's still another problem end users confuse the huge delay of commit job (sometimes hours) with being stuck unless they took the stack trace. Unfortunately Hadoop layer leaves log messages with DEBUG level, hence no log message may be written during commit phase. (I've added latency information on DEBUG log message in Hadoop side via MAPREDUCE-7317, but will be available in 3.3.1+ and the log level is still DEBUG.)
Probably we'd need to print out some informative log message during commit phase periodically - not sure where is the right place to fix, Hadoop vs Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773871847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134906/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570209352
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773374300
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774960234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134999/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774838774
**[Test build #134999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134999/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774828686
retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570209231
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
yea, that's more accurate.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Updated.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
Followed this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570209231
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
yea, that's more accurate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959
Actually I have been thinking about this - while I think this helps to track down the elapsed time on committing job, there's still another problem end users confuse the huge delay of commit job (sometimes hours) with being stuck unless they took the stack trace. Unfortunately Hadoop layer leaves log messages with DEBUG level, hence no log message may be written during commit phase. (I've added latency information on DEBUG log message in Hadoop side via MAPREDUCE-7317, but will be available in 3.3.1+ and the log level is still DEBUG.)
Probably we'd need to print out some informative log message during commit phase periodically - not sure where is the right place to fix, Hadoop vs Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773374300
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570646739
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
+1 for @yaooqinn 's suggestion to use `Utils.timeTakenMs`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r571531720
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,9 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
- committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Start to commit write Job ${description.uuid}.")
+ val (_, duration) = Utils.timeTakenMs { committer.commitJob(job, commitMsgs) }
+ logInfo(s"Write Job ${description.uuid} committed. Elapsed time: $duration ms.")
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773785733
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39488/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-775020058
thanks all !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773867185
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134899/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570179196
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Cost time -> Elapsed time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773255967
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39455/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773767528
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773349317
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773734590
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773848350
**[Test build #134899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134899/testReport)** for PR 31471 at commit [`ff2cce2`](https://github.com/apache/spark/commit/ff2cce269c49b0fed664b2caa4101560217619e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774960234
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134999/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570211414
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
How about use ` val (_, duration) = Utils.timeTakenMs { committer.commitJob(job, commitMsgs) }`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773499549
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134873/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773743688
**[Test build #134899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134899/testReport)** for PR 31471 at commit [`ff2cce2`](https://github.com/apache/spark/commit/ff2cce269c49b0fed664b2caa4101560217619e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773773206
@HeartSaVioR thanks for the sharing. Yea, agree. It's better to provide a progress-like stuff during commit job, but seems it's hard to do this at Spark side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773223589
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39455/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773187923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774942511
Thanks! Merging to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774838774
**[Test build #134999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134999/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773306903
**[Test build #134873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134873/testReport)** for PR 31471 at commit [`185ec52`](https://github.com/apache/spark/commit/185ec522c444a399d64239b7c5e63c53b175640c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #31471:
URL: https://github.com/apache/spark/pull/31471
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773187923
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773240198
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39455/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773790363
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39488/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774828933
Will merge once either Jenkins or GA passes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774854297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39582/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774854297
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39582/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773255967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773772266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773772266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39483/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773773206
@HeartSaVioR thanks for the sharing. Yea, agree. It's better to provide a progress-like stuff during commit job, but seems it's hard to do this at Spark side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773858199
OK. I'll leave this till early next week and merge if there's no further comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773867185
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134899/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773255967
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39455/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774853639
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39582/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773329865
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39459/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r571531720
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,9 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
- committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Start to commit write Job ${description.uuid}.")
+ val (_, duration) = Utils.timeTakenMs { committer.commitJob(job, commitMsgs) }
+ logInfo(s"Write Job ${description.uuid} committed. Elapsed time: $duration ms.")
Review comment:
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570096278
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Maybe should assign `System.nanoTime()` to a `val commitJobEndTime` first, otherwise the `Cost time` may not be accurate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773499549
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134873/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570179196
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Cost time -> Elapsed time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773495836
**[Test build #134873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134873/testReport)** for PR 31471 at commit [`185ec52`](https://github.com/apache/spark/commit/185ec522c444a399d64239b7c5e63c53b175640c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773773238
**[Test build #134906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134906/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773306903
**[Test build #134873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134873/testReport)** for PR 31471 at commit [`185ec52`](https://github.com/apache/spark/commit/185ec522c444a399d64239b7c5e63c53b175640c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773870939
**[Test build #134906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134906/testReport)** for PR 31471 at commit [`9d6eec7`](https://github.com/apache/spark/commit/9d6eec760927d7ae01c7a4b0f0fb6457df80ce6f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570096278
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,11 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ logInfo(s"Write Job ${description.uuid} committed. " +
+ s"Cost time: ${(System.nanoTime() - commitJobStartTime) / 1000 / 1000} ms")
Review comment:
Maybe should assign `System.nanoTime()` to a `val commitJobEndTime` first, otherwise the `Cost time` may not be accurate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-774852374
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39582/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570646739
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
+1 for @yaooqinn 's suggestion to use `Utils.timeTakenMs`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773790363
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39488/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773347944
**[Test build #134868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134868/testReport)** for PR 31471 at commit [`def94f1`](https://github.com/apache/spark/commit/def94f16f0704c98c69c2cbd62e4ac3229619180).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570211414
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
How about use ` val (_, duration) = Utils.timeTakenMs { committer.commitJob(job, commitMsgs) }`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #31471:
URL: https://github.com/apache/spark/pull/31471#discussion_r570650285
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -217,8 +217,12 @@ object FileFormatWriter extends Logging {
val commitMsgs = ret.map(_.commitMsg)
+ val commitJobStartTime = System.nanoTime()
+ logInfo(s"Start to commit write Job ${description.uuid}.")
committer.commitJob(job, commitMsgs)
- logInfo(s"Write Job ${description.uuid} committed.")
+ val commitJobEndTime = System.nanoTime()
+ logInfo(s"Write Job ${description.uuid} committed. " +
Review comment:
Followed this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959
Actually I have been thinking about this - while I think this helps to track down the elapsed time on committing job, there's still another problem end users confuse the huge delay of commit job with being stuck unless they took the stack trace. Unfortunately Hadoop layer leaves log messages with DEBUG level, hence no log message may be written during commit phase. (I've added latency information on DEBUG log message in Hadoop side via MAPREDUCE-7317, but will be available in 3.3.1+ and the log level is still DEBUG.)
Probably we'd need to print out some informative log message during commit phase periodically - not sure where is the right place to fix, Hadoop vs Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773871847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134906/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773187923
**[Test build #134868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134868/testReport)** for PR 31471 at commit [`def94f1`](https://github.com/apache/spark/commit/def94f16f0704c98c69c2cbd62e4ac3229619180).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773787166
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39488/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773743688
**[Test build #134899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134899/testReport)** for PR 31471 at commit [`ff2cce2`](https://github.com/apache/spark/commit/ff2cce269c49b0fed664b2caa4101560217619e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org