You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2025/03/19 10:09:41 UTC
(spark) branch master updated: [MINOR][SQL][TESTS] Ignore messages below error level from `HadoopFSUtils` for sql module tests
This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f4493fd00ff7 [MINOR][SQL][TESTS] Ignore messages below error level from `HadoopFSUtils` for sql module tests
f4493fd00ff7 is described below
commit f4493fd00ff781b115b7f3bc3144ed06ae62dcf4
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Wed Mar 19 18:09:30 2025 +0800
[MINOR][SQL][TESTS] Ignore messages below error level from `HadoopFSUtils` for sql module tests
### What changes were proposed in this pull request?
This pr ignore messages below error level from `HadoopFSUtils` for sql module tests.
### Why are the changes needed?
From the test logs of Github Actions, there are a large number of logs from `HadoopFSUtils`. For example, in the pipeline named `sql-other tests`, there are approximately 40,000 similar logs like the following:
- https://github.com/apache/spark/actions/runs/13915102741/job/38936641387
```
04:22:51.192 WARN org.apache.spark.util.HadoopFSUtils: The directory /home/runner/work/spark/spark/target/tmp/spark-e3d8a446-2dd8-42fd-a64f-b96504d8cd69/deleted was not found. Was it deleted very recently?
04:22:51.193 WARN org.apache.spark.util.HadoopFSUtils: The directory /home/runner/work/spark/spark/target/tmp/spark-e3d8a446-2dd8-42fd-a64f-b96504d8cd69/deleted was not found. Was it deleted very recently?
[info] - InMemoryFileIndex: root folders that don't exist don't throw exceptions (6 milliseconds)
04:22:51.244 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.271 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.280 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.281 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.307 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.332 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.334 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
04:22:51.334 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently?
```
Similar logs are more like noise, so the current pr makes `HadoopFSUtils` only print error-level logs during SQL module testing.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass Github Actions
- Manual inspection shows that the new tests in the SQL module no longer print similar logs as mentioned above.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50316 from LuciferYang/sql-log-level.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: yangjie01 <ya...@baidu.com>
---
sql/core/src/test/resources/log4j2.properties | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/sql/core/src/test/resources/log4j2.properties b/sql/core/src/test/resources/log4j2.properties
index fdbc35e70ab4..9b0e2af355fd 100644
--- a/sql/core/src/test/resources/log4j2.properties
+++ b/sql/core/src/test/resources/log4j2.properties
@@ -84,3 +84,7 @@ logger.structured.name = org.apache.spark.sql.LogQuerySuite
logger.structured.level = trace
logger.structured.appenderRefs = structured
logger.structured.appenderRef.structured.ref = structured
+
+# Ignore messages below error level from HadoopFSUtils, because it's a bit verbose
+logger.hadoopfsutils.name = org.apache.spark.util.HadoopFSUtils
+logger.hadoopfsutils.level = error
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org