[spark] branch master updated: [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 2fc9c0b  [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker
2fc9c0b is described below

commit 2fc9c0bfb5d43a6ee1dbbf941b84e4c3dd74d8ef
Author: tooptoop4 <33...@users.noreply.github.com>
AuthorDate: Thu Aug 19 10:31:10 2021 +0900

    [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker
    
    ### What changes were proposed in this pull request?
    
    This log should at least be WARN not INFO (in org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala )
    "Expected $numSubmittedFiles files, but only saw $numFiles."
    
    ### Why are the changes needed?
    
    INFO logs don't indicate possible issue but WARN logs should
    
    ### Does this PR introduce any user-facing change?
    
    Yes, Log level changed.
    
    ### How was this patch tested?
    
    manual, trivial change
    
    Closes #33332 from tooptoop4/warn.
    
    Authored-by: tooptoop4 <33...@users.noreply.github.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 .../apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
index f3815ab..a8fae66 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
@@ -169,7 +169,7 @@ class BasicWriteTaskStatsTracker(
     }
 
     if (numSubmittedFiles != numFiles) {
-      logInfo(s"Expected $numSubmittedFiles files, but only saw $numFiles. " +
+      logWarning(s"Expected $numSubmittedFiles files, but only saw $numFiles. " +
         "This could be due to the output format not writing empty files, " +
         "or files being not immediately visible in the filesystem.")
     }

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org