[GitHub] [spark] HeartSaVioR commented on pull request #31471: [SPARK-34355][SQL] Add log and time cost for commit job

[GitBox <gi...@apache.org>]

HeartSaVioR commented on pull request #31471:
URL: https://github.com/apache/spark/pull/31471#issuecomment-773744959


   Actually I have been thinking about this - while I think this helps to track down the elapsed time on committing job, there's still another problem end users confuse the huge delay of commit job with being stuck unless they took the stack trace. Unfortunately Hadoop layer leaves log messages with DEBUG level, hence no log message may be written during commit phase. (I've added latency information on DEBUG log message in Hadoop side via MAPREDUCE-7317, but will be available in 3.3.1+ and the log level is still DEBUG.)
   
   Probably we'd need to print out some informative log message during commit phase periodically - not sure where is the right place to fix, Hadoop vs Spark.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org