You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2023/08/23 05:45:35 UTC
[spark] branch branch-3.4 updated: [SPARK-44922][TESTS] Disable o.a.p.h.InternalParquetRecordWriter logs for tests
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 6496ca14a0d [SPARK-44922][TESTS] Disable o.a.p.h.InternalParquetRecordWriter logs for tests
6496ca14a0d is described below
commit 6496ca14a0d3f500b1cb51ed312fa4bee75e043c
Author: Kent Yao <ya...@apache.org>
AuthorDate: Wed Aug 23 13:41:25 2023 +0800
[SPARK-44922][TESTS] Disable o.a.p.h.InternalParquetRecordWriter logs for tests
### What changes were proposed in this pull request?
This PR disable InternalParquetRecordWriter logs for SQL tests
### Why are the changes needed?
The InternalParquetRecordWriter creates over 1,800 records, which equates to more than 80,000 lines of code. This accounts for 80% of the volume of the "slow tests" before the GitHub action truncates it.
#### A record for example
```
2023-08-22T13:38:14.5103112Z 13:38:14.406 WARN org.apache.parquet.hadoop.InternalParquetRecordWriter: Too much memory used: Store {
2023-08-22T13:38:14.5103259Z [dummy_col] optional int32 dummy_col {
2023-08-22T13:38:14.5103352Z r:0 bytes
2023-08-22T13:38:14.5103444Z d:0 bytes
2023-08-22T13:38:14.5103580Z data: FallbackValuesWriter{
2023-08-22T13:38:14.5103739Z data: initial: DictionaryValuesWriter{
2023-08-22T13:38:14.5103854Z data: initial: dict:8
2023-08-22T13:38:14.5103976Z data: initial: values:400
2023-08-22T13:38:14.5104079Z data: initial:}
2023-08-22T13:38:14.5104087Z
2023-08-22T13:38:14.5104325Z data: fallback: PLAIN CapacityByteArrayOutputStream 0 slabs, 0 bytes
2023-08-22T13:38:14.5104418Z data:}
2023-08-22T13:38:14.5104426Z
2023-08-22T13:38:14.5104695Z pages: ColumnChunkPageWriter ConcatenatingByteArrayCollector 0 slabs, 0 bytes
2023-08-22T13:38:14.5104795Z total: 400/408
2023-08-22T13:38:14.5104936Z }
2023-08-22T13:38:14.5105129Z [expected_rowIdx_col] optional int32 expected_rowIdx_col {
2023-08-22T13:38:14.5105291Z r:0 bytes
2023-08-22T13:38:14.5105383Z d:0 bytes
2023-08-22T13:38:14.5105518Z data: FallbackValuesWriter{
2023-08-22T13:38:14.5105679Z data: initial: DictionaryValuesWriter{
2023-08-22T13:38:14.5105797Z data: initial: dict:400
2023-08-22T13:38:14.5105919Z data: initial: values:400
2023-08-22T13:38:14.5106022Z data: initial:}
2023-08-22T13:38:14.5106030Z
2023-08-22T13:38:14.5106267Z data: fallback: PLAIN CapacityByteArrayOutputStream 0 slabs, 0 bytes
2023-08-22T13:38:14.5106356Z data:}
2023-08-22T13:38:14.5106364Z
2023-08-22T13:38:14.5106636Z pages: ColumnChunkPageWriter ConcatenatingByteArrayCollector 0 slabs, 0 bytes
2023-08-22T13:38:14.5106736Z total: 400/800
2023-08-22T13:38:14.5106820Z }
2023-08-22T13:38:14.5106942Z [id] required int64 id {
2023-08-22T13:38:14.5107037Z r:0 bytes
2023-08-22T13:38:14.5107133Z d:0 bytes
2023-08-22T13:38:14.5107275Z data: FallbackValuesWriter{
2023-08-22T13:38:14.5107436Z data: initial: DictionaryValuesWriter{
2023-08-22T13:38:14.5107557Z data: initial: dict:800
2023-08-22T13:38:14.5107677Z data: initial: values:400
2023-08-22T13:38:14.5107779Z data: initial:}
2023-08-22T13:38:14.5107787Z
2023-08-22T13:38:14.5108022Z data: fallback: PLAIN CapacityByteArrayOutputStream 0 slabs, 0 bytes
2023-08-22T13:38:14.5108113Z data:}
2023-08-22T13:38:14.5108121Z
2023-08-22T13:38:14.5108384Z pages: ColumnChunkPageWriter ConcatenatingByteArrayCollector 0 slabs, 0 bytes
2023-08-22T13:38:14.5108485Z total: 800/1,200
2023-08-22T13:38:14.5108570Z }
2023-08-22T13:38:14.5108655Z }
2023-08-22T13:38:14.5108664Z
```
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
see log volume of SQL tests
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #42614 from yaooqinn/log.
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
(cherry picked from commit 34b51cc784478f65c4c9a6cf3fdc0ed4ca31f745)
Signed-off-by: Kent Yao <ya...@apache.org>
---
connector/connect/server/src/test/resources/log4j2.properties | 5 +++++
sql/core/src/test/resources/log4j2.properties | 5 +++++
sql/hive-thriftserver/src/test/resources/log4j2.properties | 5 +++++
sql/hive/src/test/resources/log4j2.properties | 5 +++++
4 files changed, 20 insertions(+)
diff --git a/connector/connect/server/src/test/resources/log4j2.properties b/connector/connect/server/src/test/resources/log4j2.properties
index ab02104c696..ac4c1de0e91 100644
--- a/connector/connect/server/src/test/resources/log4j2.properties
+++ b/connector/connect/server/src/test/resources/log4j2.properties
@@ -37,3 +37,8 @@ appender.console.layout.pattern = %t: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
logger.jetty.level = warn
+
+# SPARK-44922: Disable o.a.p.h.InternalParquetRecordWriter logs for tests
+logger.parquet_recordwriter.name = org.apache.parquet.hadoop.InternalParquetRecordWriter
+logger.parquet_recordwriter.additivity = false
+logger.parquet_recordwriter.level = off
diff --git a/sql/core/src/test/resources/log4j2.properties b/sql/core/src/test/resources/log4j2.properties
index b1f3a726a92..49df23c4fc7 100644
--- a/sql/core/src/test/resources/log4j2.properties
+++ b/sql/core/src/test/resources/log4j2.properties
@@ -45,6 +45,11 @@ logger.parquet_recordreader.name = org.apache.parquet.hadoop.ParquetRecordReader
logger.parquet_recordreader.additivity = false
logger.parquet_recordreader.level = off
+# SPARK-44922: Disable o.a.p.h.InternalParquetRecordWriter logs for tests
+logger.parquet_recordwriter.name = org.apache.parquet.hadoop.InternalParquetRecordWriter
+logger.parquet_recordwriter.additivity = false
+logger.parquet_recordwriter.level = off
+
logger.parquet_outputcommitter.name = org.apache.parquet.hadoop.ParquetOutputCommitter
logger.parquet_outputcommitter.additivity = false
logger.parquet_outputcommitter.level = off
diff --git a/sql/hive-thriftserver/src/test/resources/log4j2.properties b/sql/hive-thriftserver/src/test/resources/log4j2.properties
index 9141d616c4a..cc483f4671e 100644
--- a/sql/hive-thriftserver/src/test/resources/log4j2.properties
+++ b/sql/hive-thriftserver/src/test/resources/log4j2.properties
@@ -61,6 +61,11 @@ logger.parquet_recordreader.name = org.apache.parquet.hadoop.ParquetRecordReader
logger.parquet_recordreader.additivity = false
logger.parquet_recordreader.level = off
+# SPARK-44922: Disable o.a.p.h.InternalParquetRecordWriter logs for tests
+logger.parquet_recordwriter.name = org.apache.parquet.hadoop.InternalParquetRecordWriter
+logger.parquet_recordwriter.additivity = false
+logger.parquet_recordwriter.level = off
+
logger.parquet_outputcommitter.name = org.apache.parquet.hadoop.ParquetOutputCommitter
logger.parquet_outputcommitter.additivity = false
logger.parquet_outputcommitter.level = off
diff --git a/sql/hive/src/test/resources/log4j2.properties b/sql/hive/src/test/resources/log4j2.properties
index 454b6905b45..6e2c01c4b14 100644
--- a/sql/hive/src/test/resources/log4j2.properties
+++ b/sql/hive/src/test/resources/log4j2.properties
@@ -45,6 +45,11 @@ logger.parquet_recordreader.name = org.apache.parquet.hadoop.ParquetRecordReader
logger.parquet_recordreader.additivity = false
logger.parquet_recordreader.level = off
+# SPARK-44922: Disable o.a.p.h.InternalParquetRecordWriter logs for tests
+logger.parquet_recordwriter.name = org.apache.parquet.hadoop.InternalParquetRecordWriter
+logger.parquet_recordwriter.additivity = false
+logger.parquet_recordwriter.level = off
+
logger.parquet_outputcommitter.name = org.apache.parquet.hadoop.ParquetOutputCommitter
logger.parquet_outputcommitter.additivity = false
logger.parquet_outputcommitter.level = off
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org