You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ec...@apache.org on 2019/07/16 13:45:15 UTC
[beam] branch spark-runner_structured-streaming updated (797b40d ->
b62a593)
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 797b40d Add PipelineResults to Spark structured streaming.
new 935e703 Update log4j configuration
new b62a593 Add spark execution plans extended debug messages.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../translation/TranslationContext.java | 1 +
runners/spark/src/main/resources/log4j.properties | 52 +++++++++++++---------
2 files changed, 33 insertions(+), 20 deletions(-)
[beam] 02/02: Add spark execution plans extended debug messages.
Posted by ec...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a commit to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b62a593d312eb9632e68c486ad6d5477d0b656a9
Author: Etienne Chauchot <ec...@apache.org>
AuthorDate: Tue Jul 16 15:32:49 2019 +0200
Add spark execution plans extended debug messages.
---
.../spark/structuredstreaming/translation/TranslationContext.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TranslationContext.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TranslationContext.java
index 4157ea6..3ca232c 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TranslationContext.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TranslationContext.java
@@ -198,6 +198,7 @@ public class TranslationContext {
dataStreamWriter.foreach(new NoOpForeachWriter<>()).start().awaitTermination();
} else {
if (options.getTestMode()) {
+ dataset.explain(true);
// cannot use dataset.show because dataset schema is binary so it will print binary
// code.
List<WindowedValue> windowedValues = ((Dataset<WindowedValue>) dataset).collectAsList();
[beam] 01/02: Update log4j configuration
Posted by ec...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a commit to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 935e70344f2d550794ecc1ec3bbfacc337fb078c
Author: Etienne Chauchot <ec...@apache.org>
AuthorDate: Tue Jul 16 11:40:36 2019 +0200
Update log4j configuration
---
runners/spark/src/main/resources/log4j.properties | 52 ++++++++++++++---------
1 file changed, 32 insertions(+), 20 deletions(-)
diff --git a/runners/spark/src/main/resources/log4j.properties b/runners/spark/src/main/resources/log4j.properties
index ec58c46..8c7800f 100644
--- a/runners/spark/src/main/resources/log4j.properties
+++ b/runners/spark/src/main/resources/log4j.properties
@@ -1,28 +1,40 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
# limitations under the License.
-################################################################################
+#
-log4j.rootLogger=OFF,console
+# Set everything to be logged to the console
+log4j.rootCategory=DEBUG, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
-log4j.appender.console.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c: %m%n
+
+# General Beam loggers
+log4j.logger.org.apache.beam.sdk=WARN
+
+# Settings to quiet third party logs that are too verbose
+log4j.logger.org.spark_project.jetty=WARN
+log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
+
+# Setting to quiet spark logs, Beam logs should standout
+log4j.logger.org.apache.beam.runners.spark=DEBUG
+log4j.logger.org.apache.spark=WARN
+log4j.logger.org.spark-project=WARN
+log4j.logger.io.netty=INFO
-# Spark structured streaming extra logging for debug purposes
-#log4j.logger.org.apache.spark=ERROR
-#log4j.logger.org.apache.beam.runners.spark.structuredstreaming.translation.TranslationContext=DEBUG
-#log4j.logger.org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator=DEBUG
+# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
+log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
+log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR