You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2022/06/02 16:29:14 UTC
[spark] branch master updated: [SPARK-39361] Don't use Log4J2's extended throwable conversion pattern in default logging configurations
This is an automated email from the ASF dual-hosted git repository.
joshrosen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd45c3656be [SPARK-39361] Don't use Log4J2's extended throwable conversion pattern in default logging configurations
fd45c3656be is described below
commit fd45c3656be6add7cf483ddfb7016b12f77d7c8e
Author: Josh Rosen <jo...@databricks.com>
AuthorDate: Thu Jun 2 09:28:34 2022 -0700
[SPARK-39361] Don't use Log4J2's extended throwable conversion pattern in default logging configurations
### What changes were proposed in this pull request?
This PR addresses a performance problem in Log4J 2 related to exception logging: in certain scenarios I observed that Log4J2's default exception stacktrace logging can be ~10x slower than Log4J 1.
The problem stems from a new log pattern format in Log4J2 called ["extended exception"](https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternExtendedException), which enriches the regular stacktrace string with information on the name of the JAR files that contained the classes in each stack frame.
Log4J queries the classloader to determine the source JAR for each class. This isn't cheap, but this information is cached and reused in future exception logging calls. In certain scenarios involving runtime-generated classes, this lookup will fail and the failed lookup result will _not_ be cached. As a result, expensive classloading operations will be performed every time such an exception is logged. In addition to being very slow, these operations take out a lock on the classloader [...]
By default, if you do not specify an explicit exception format in your logging pattern then Log4J2 will add this "extended exception" pattern (see PatternLayout's alwaysWriteExceptions flag in Log4J's documentation, plus [the code implementing that flag](https://github.com/apache/logging-log4j2/blob/d6c8ab0863c551cdf0f8a5b1966ab45e3cddf572/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/PatternParser.java#L206-L209) in Log4J2).
In this PR, I have updated Spark's default Log4J2 configurations so that each pattern layout includes an explicit %ex so that it uses the normal (non-extended) exception logging format. This is the workaround that is currently recommended on the Log4J JIRA.
### Why are the changes needed?
Avoid performance regressions in Spark programs which use Spark's default Log4J 2 configuration and log many exceptions. Although it's true that any program logging exceptions at a high rate should probably just fix the source of the exceptions, I think it's still a good idea for us to try to fix this out-of-the-box performance difference so that users' existing workloads do not regress when upgrading to 3.3.0.
### Does this PR introduce _any_ user-facing change?
Yes: it changes the default exception logging format so that it matches Log4J 1's default rather than Log4J 2's. The new format is consistent with behavior in previous Spark versions, but is different than the behavior in the current Spark 3.3.0-rc3.
### How was this patch tested?
Existing tests.
Closes #36747 from JoshRosen/disable-log4j2-extended-exception-pattern.
Authored-by: Josh Rosen <jo...@databricks.com>
Signed-off-by: Josh Rosen <jo...@databricks.com>
---
R/log4j2.properties | 2 +-
common/kvstore/src/test/resources/log4j2.properties | 2 +-
common/network-common/src/test/resources/log4j2.properties | 2 +-
common/network-shuffle/src/test/resources/log4j2.properties | 2 +-
conf/log4j2.properties.template | 8 +++++++-
connector/avro/src/test/resources/log4j2.properties | 2 +-
.../docker-integration-tests/src/test/resources/log4j2.properties | 4 ++--
connector/kafka-0-10-sql/src/test/resources/log4j2.properties | 2 +-
.../src/test/resources/log4j2.properties | 2 +-
connector/kafka-0-10/src/test/resources/log4j2.properties | 2 +-
connector/kinesis-asl/src/main/resources/log4j2.properties | 4 ++--
connector/kinesis-asl/src/test/resources/log4j2.properties | 2 +-
.../main/resources/org/apache/spark/log4j2-defaults.properties | 2 +-
core/src/main/scala/org/apache/spark/TestUtils.scala | 2 +-
.../main/scala/org/apache/spark/util/logging/DriverLogger.scala | 2 +-
core/src/test/resources/log4j2.properties | 4 ++--
docs/configuration.md | 2 +-
graphx/src/test/resources/log4j2.properties | 2 +-
hadoop-cloud/src/test/resources/log4j2.properties | 4 ++--
launcher/src/test/resources/log4j2.properties | 4 ++--
mllib/src/test/resources/log4j2.properties | 2 +-
repl/src/test/resources/log4j2.properties | 2 +-
repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala | 2 +-
.../kubernetes/core/src/test/resources/log4j2.properties | 2 +-
.../src/test/resources/log-config-test-log4j.properties | 2 +-
.../integration-tests/src/test/resources/log4j2.properties | 2 +-
.../spark/deploy/k8s/integrationtest/DecommissionSuite.scala | 2 +-
resource-managers/mesos/src/test/resources/log4j2.properties | 2 +-
resource-managers/yarn/src/test/resources/log4j2.properties | 2 +-
.../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 2 +-
sql/catalyst/src/test/resources/log4j2.properties | 2 +-
sql/core/src/test/resources/log4j2.properties | 4 ++--
.../main/java/org/apache/hive/service/cli/CLIServiceUtils.java | 4 ++--
sql/hive-thriftserver/src/test/resources/log4j2.properties | 4 ++--
.../spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala | 2 +-
.../org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala | 2 +-
sql/hive/src/test/resources/log4j2.properties | 4 ++--
streaming/src/test/resources/log4j2.properties | 2 +-
38 files changed, 53 insertions(+), 47 deletions(-)
diff --git a/R/log4j2.properties b/R/log4j2.properties
index 8ed7b9f6c8c..689518e4280 100644
--- a/R/log4j2.properties
+++ b/R/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.eclipse.jetty
diff --git a/common/kvstore/src/test/resources/log4j2.properties b/common/kvstore/src/test/resources/log4j2.properties
index 9a0fd7cdc6f..551abd8413c 100644
--- a/common/kvstore/src/test/resources/log4j2.properties
+++ b/common/kvstore/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Silence verbose logs from 3rd-party libraries.
logger.netty.name = io.netty
diff --git a/common/network-common/src/test/resources/log4j2.properties b/common/network-common/src/test/resources/log4j2.properties
index 9a0fd7cdc6f..551abd8413c 100644
--- a/common/network-common/src/test/resources/log4j2.properties
+++ b/common/network-common/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Silence verbose logs from 3rd-party libraries.
logger.netty.name = io.netty
diff --git a/common/network-shuffle/src/test/resources/log4j2.properties b/common/network-shuffle/src/test/resources/log4j2.properties
index 4fc8e41b6f0..fbdac2fb170 100644
--- a/common/network-shuffle/src/test/resources/log4j2.properties
+++ b/common/network-shuffle/src/test/resources/log4j2.properties
@@ -24,4 +24,4 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
diff --git a/conf/log4j2.properties.template b/conf/log4j2.properties.template
index 99f68a8a9e9..ab96e03baed 100644
--- a/conf/log4j2.properties.template
+++ b/conf/log4j2.properties.template
@@ -19,11 +19,17 @@
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console
+# In the pattern layout configuration below, we specify an explicit `%ex` conversion
+# pattern for logging Throwables. If this was omitted, then (by default) Log4J would
+# implicitly add an `%xEx` conversion pattern which logs stacktraces with additional
+# class packaging information. That extra information can sometimes add a substantial
+# performance overhead, so we disable it in our default logging config.
+# For more information, see SPARK-39361.
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
# Set the default spark-shell/spark-sql log level to WARN. When running the
# spark-shell/spark-sql, the log level for these classes is used to overwrite
diff --git a/connector/avro/src/test/resources/log4j2.properties b/connector/avro/src/test/resources/log4j2.properties
index 31a235c5d82..02746f58f46 100644
--- a/connector/avro/src/test/resources/log4j2.properties
+++ b/connector/avro/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
diff --git a/connector/docker-integration-tests/src/test/resources/log4j2.properties b/connector/docker-integration-tests/src/test/resources/log4j2.properties
index a6db7b73f6f..bc2edf5a771 100644
--- a/connector/docker-integration-tests/src/test/resources/log4j2.properties
+++ b/connector/docker-integration-tests/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -33,7 +33,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/connector/kafka-0-10-sql/src/test/resources/log4j2.properties b/connector/kafka-0-10-sql/src/test/resources/log4j2.properties
index 4c2d2d0b053..cb454ef0498 100644
--- a/connector/kafka-0-10-sql/src/test/resources/log4j2.properties
+++ b/connector/kafka-0-10-sql/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
diff --git a/connector/kafka-0-10-token-provider/src/test/resources/log4j2.properties b/connector/kafka-0-10-token-provider/src/test/resources/log4j2.properties
index 31a235c5d82..02746f58f46 100644
--- a/connector/kafka-0-10-token-provider/src/test/resources/log4j2.properties
+++ b/connector/kafka-0-10-token-provider/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
diff --git a/connector/kafka-0-10/src/test/resources/log4j2.properties b/connector/kafka-0-10/src/test/resources/log4j2.properties
index 31a235c5d82..02746f58f46 100644
--- a/connector/kafka-0-10/src/test/resources/log4j2.properties
+++ b/connector/kafka-0-10/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
diff --git a/connector/kinesis-asl/src/main/resources/log4j2.properties b/connector/kinesis-asl/src/main/resources/log4j2.properties
index 0c0904b088b..9538957d5dd 100644
--- a/connector/kinesis-asl/src/main/resources/log4j2.properties
+++ b/connector/kinesis-asl/src/main/resources/log4j2.properties
@@ -23,14 +23,14 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %p %c{1}: %m%n%ex
# Console appender
appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
# Settings to quiet third party logs that are too verbose
logger.jetty1.name = org.sparkproject.jetty
diff --git a/connector/kinesis-asl/src/test/resources/log4j2.properties b/connector/kinesis-asl/src/test/resources/log4j2.properties
index 08f43461b96..5f89859463a 100644
--- a/connector/kinesis-asl/src/test/resources/log4j2.properties
+++ b/connector/kinesis-asl/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/core/src/main/resources/org/apache/spark/log4j2-defaults.properties b/core/src/main/resources/org/apache/spark/log4j2-defaults.properties
index cb3c70e2b87..62eab7f3ef9 100644
--- a/core/src/main/resources/org/apache/spark/log4j2-defaults.properties
+++ b/core/src/main/resources/org/apache/spark/log4j2-defaults.properties
@@ -23,7 +23,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
# Settings to quiet third party logs that are too verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/core/src/main/scala/org/apache/spark/TestUtils.scala b/core/src/main/scala/org/apache/spark/TestUtils.scala
index 104e98b8ae0..880f8856fe3 100644
--- a/core/src/main/scala/org/apache/spark/TestUtils.scala
+++ b/core/src/main/scala/org/apache/spark/TestUtils.scala
@@ -431,7 +431,7 @@ private[spark] object TestUtils {
val appenderBuilder = builder.newAppender("console", "CONSOLE")
.addAttribute("target", ConsoleAppender.Target.SYSTEM_ERR)
appenderBuilder.add(builder.newLayout("PatternLayout")
- .addAttribute("pattern", "%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n"))
+ .addAttribute("pattern", "%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex"))
builder.add(appenderBuilder)
builder.add(builder.newRootLogger(level).add(builder.newAppenderRef("console")))
val configuration = builder.build()
diff --git a/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala b/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala
index 4854a84eb56..c826cef213f 100644
--- a/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala
+++ b/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala
@@ -42,7 +42,7 @@ private[spark] class DriverLogger(conf: SparkConf) extends Logging {
private val UPLOAD_CHUNK_SIZE = 1024 * 1024
private val UPLOAD_INTERVAL_IN_SECS = 5
- private val DEFAULT_LAYOUT = "%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n"
+ private val DEFAULT_LAYOUT = "%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex"
private val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770", 8).toShort)
private val localLogFile: String = FileUtils.getFile(
diff --git a/core/src/test/resources/log4j2.properties b/core/src/test/resources/log4j2.properties
index c6cd10d639e..ab02104c696 100644
--- a/core/src/test/resources/log4j2.properties
+++ b/core/src/test/resources/log4j2.properties
@@ -23,7 +23,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n
+appender.console.layout.pattern = %t: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/docs/configuration.md b/docs/configuration.md
index 78a9fb33b02..22f16b8132e 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -449,7 +449,7 @@ of the most common options to set are:
</tr>
<tr>
<td><code>spark.driver.log.layout</code></td>
- <td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n</td>
+ <td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex</td>
<td>
The layout for the driver logs that are synced to <code>spark.driver.log.dfsDir</code>. If this is not configured,
it uses the layout for the first appender defined in log4j2.properties. If that is also not configured, driver logs
diff --git a/graphx/src/test/resources/log4j2.properties b/graphx/src/test/resources/log4j2.properties
index 08f43461b96..5f89859463a 100644
--- a/graphx/src/test/resources/log4j2.properties
+++ b/graphx/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/hadoop-cloud/src/test/resources/log4j2.properties b/hadoop-cloud/src/test/resources/log4j2.properties
index 640ec2e6305..01a9cafafa8 100644
--- a/hadoop-cloud/src/test/resources/log4j2.properties
+++ b/hadoop-cloud/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -33,7 +33,7 @@ appender.console.type = Console
appender.console.name = STDERR
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %t: %m%n
+appender.console.layout.pattern = %t: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark_project.jetty
diff --git a/launcher/src/test/resources/log4j2.properties b/launcher/src/test/resources/log4j2.properties
index f7f2d7cfe4c..62d38fbf8f7 100644
--- a/launcher/src/test/resources/log4j2.properties
+++ b/launcher/src/test/resources/log4j2.properties
@@ -23,13 +23,13 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
appender.childproc.type = Console
appender.childproc.name = childproc
appender.childproc.target = SYSTEM_ERR
appender.childproc.layout.type = PatternLayout
-appender.childproc.layout.pattern = %t: %m%n
+appender.childproc.layout.pattern = %t: %m%n%ex
appender.outputredirtest.type = LogAppender
appender.outputredirtest.name = outputredirtest
diff --git a/mllib/src/test/resources/log4j2.properties b/mllib/src/test/resources/log4j2.properties
index 101a732ea97..4c0b6e67527 100644
--- a/mllib/src/test/resources/log4j2.properties
+++ b/mllib/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/repl/src/test/resources/log4j2.properties b/repl/src/test/resources/log4j2.properties
index 2654a3983a6..b644dd5f7b9 100644
--- a/repl/src/test/resources/log4j2.properties
+++ b/repl/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = file
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
index 2ff2120625b..69e1273f5fa 100644
--- a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
+++ b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
@@ -285,7 +285,7 @@ class ReplSuite extends SparkFunSuite with BeforeAndAfterAll {
|appender.console.target = SYSTEM_ERR
|appender.console.follow = true
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+ |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|
|# Set the log level for this class to ERROR same as the default setting.
|logger.repl.name = org.apache.spark.repl.Main
diff --git a/resource-managers/kubernetes/core/src/test/resources/log4j2.properties b/resource-managers/kubernetes/core/src/test/resources/log4j2.properties
index 712e6d6c30c..210405374f0 100644
--- a/resource-managers/kubernetes/core/src/test/resources/log4j2.properties
+++ b/resource-managers/kubernetes/core/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
diff --git a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
index 17b8d598ac6..4aeea886251 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
+++ b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties
@@ -23,4 +23,4 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n%ex
diff --git a/resource-managers/kubernetes/integration-tests/src/test/resources/log4j2.properties b/resource-managers/kubernetes/integration-tests/src/test/resources/log4j2.properties
index 75ac84ac6e1..e7680d87bdf 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/resources/log4j2.properties
+++ b/resource-managers/kubernetes/integration-tests/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/integration-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
index 51ea1307236..5d1a57fb46e 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
+++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
@@ -47,7 +47,7 @@ private[spark] trait DecommissionSuite { k8sSuite: KubernetesSuite =>
|appender.console.name = console
|appender.console.target = SYSTEM_OUT
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+ |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|
|logger.spark.name = org.apache.spark
|logger.spark.level = debug
diff --git a/resource-managers/mesos/src/test/resources/log4j2.properties b/resource-managers/mesos/src/test/resources/log4j2.properties
index a0d309ccb1d..88847106120 100644
--- a/resource-managers/mesos/src/test/resources/log4j2.properties
+++ b/resource-managers/mesos/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/resource-managers/yarn/src/test/resources/log4j2.properties b/resource-managers/yarn/src/test/resources/log4j2.properties
index 96107fb3a3a..aa93bc14552 100644
--- a/resource-managers/yarn/src/test/resources/log4j2.properties
+++ b/resource-managers/yarn/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index f92bcdd6775..87ea44255cc 100644
--- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -53,7 +53,7 @@ abstract class BaseYarnClusterSuite
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+ |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|logger.jetty.name = org.sparkproject.jetty
|logger.jetty.level = warn
|logger.eclipse.name = org.eclipse.jetty
diff --git a/sql/catalyst/src/test/resources/log4j2.properties b/sql/catalyst/src/test/resources/log4j2.properties
index 08f43461b96..5f89859463a 100644
--- a/sql/catalyst/src/test/resources/log4j2.properties
+++ b/sql/catalyst/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
diff --git a/sql/core/src/test/resources/log4j2.properties b/sql/core/src/test/resources/log4j2.properties
index 2ab43f896fc..b1f3a726a92 100644
--- a/sql/core/src/test/resources/log4j2.properties
+++ b/sql/core/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn
@@ -34,7 +34,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Set the logger level of File Appender to WARN
appender.file.filter.threshold.type = ThresholdFilter
diff --git a/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIServiceUtils.java b/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIServiceUtils.java
index a371b137170..97ea1ca0d4c 100644
--- a/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIServiceUtils.java
+++ b/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIServiceUtils.java
@@ -29,9 +29,9 @@ public class CLIServiceUtils {
private static final char SEARCH_STRING_ESCAPE = '\\';
public static final StringLayout verboseLayout = PatternLayout.newBuilder().withPattern(
- "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n").build();
+ "%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n%ex").build();
public static final StringLayout nonVerboseLayout = PatternLayout.newBuilder().withPattern(
- "%-5p : %m%n").build();
+ "%-5p : %m%n%ex").build();
/**
* Convert a SQL search pattern into an equivalent Java Regex.
diff --git a/sql/hive-thriftserver/src/test/resources/log4j2.properties b/sql/hive-thriftserver/src/test/resources/log4j2.properties
index 939335bf3ac..5a3681a2a7e 100644
--- a/sql/hive-thriftserver/src/test/resources/log4j2.properties
+++ b/sql/hive-thriftserver/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
appender.console.filter.1.type = Filters
@@ -43,7 +43,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
appender.file.filter.1.type = Filters
diff --git a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
index d135fe7efbb..7c93fa5ab2e 100644
--- a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
+++ b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
@@ -1228,7 +1228,7 @@ abstract class HiveThriftServer2TestBase extends SparkFunSuite with BeforeAndAft
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+ |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
diff --git a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
index 5d94be38de0..cd5bb1f6283 100644
--- a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
+++ b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala
@@ -80,7 +80,7 @@ class UISeleniumSuite
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
- |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+ |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
diff --git a/sql/hive/src/test/resources/log4j2.properties b/sql/hive/src/test/resources/log4j2.properties
index 16b89247891..cf9be6c68a5 100644
--- a/sql/hive/src/test/resources/log4j2.properties
+++ b/sql/hive/src/test/resources/log4j2.properties
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
-appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
+appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn
@@ -34,7 +34,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Set the logger level of File Appender to WARN
appender.file.filter.threshold.type = ThresholdFilter
diff --git a/streaming/src/test/resources/log4j2.properties b/streaming/src/test/resources/log4j2.properties
index 08f43461b96..5f89859463a 100644
--- a/streaming/src/test/resources/log4j2.properties
+++ b/streaming/src/test/resources/log4j2.properties
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
-appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
+appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org