You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "vidit Singh (JIRA)" <ji...@apache.org> on 2016/12/23 07:28:58 UTC
[jira] [Created] (SPARK-18987) I am trying to disable Spark Stage
progress logs on cluster.
vidit Singh created SPARK-18987:
-----------------------------------
Summary: I am trying to disable Spark Stage progress logs on cluster.
Key: SPARK-18987
URL: https://issues.apache.org/jira/browse/SPARK-18987
Project: Spark
Issue Type: Question
Components: Input/Output
Environment: Hadoop Cluster
Reporter: vidit Singh
I am trying to disable Stage progress of spark on Cluster. As i don't want to see those stages. I have tried every possible things to disable that but still that problem persist. I have tried the following things : Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);
Even modified log4j to log4j.logger.org.apache.spark=OFF
I have tried passing spark.ui.showConsoleProgress=false in Spark Submit query. even passing external log4j along with spark submit didn't work for me. : --files log4j.properties .
Output :
[Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 11:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2][Stage 20:==> (1 + 1) / 2]
[Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2]
[Stage 8:=========> (1 + 1) / 2][Stage 15:=========> (1 + 1) / 2]
[Stage 15:=============================> (1 + 1) / 2]
[Stage 10:> (0 + 180) / 200]
[Stage 10:> (0 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (2 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (5 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (14 + 180) / 200][Stage 17:> (0 + 0) / 200]
I don't want to see these stages in my logs. Please help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org