You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by piaozhexiu <gi...@git.apache.org> on 2015/04/05 07:59:04 UTC
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
GitHub user piaozhexiu opened a pull request:
https://github.com/apache/spark/pull/5362
[SPARK-6712][YARN] Allow lower the log level in YARN client while keeping AM tracking URL printed
In YARN mode, log messages are quite verbose in interactive shells (spark-shell, spark-sql, pyspark), and they sometimes mingle with shell prompts. In fact, it's very easy to tone it down via log4j.properties, but the problem is that the AM tracking URL is not printed if I do that.
It would be nice if I could keep the AM tracking URL while disabling the other INFO messages that don't matter to most end users.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/piaozhexiu/spark SPARK-6712
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5362.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5362
----
commit 9b0a047e1e34e351f22329156efb50f4a452e091
Author: Cheolsoo Park <ch...@netflix.com>
Date: 2015-04-05T05:57:05Z
Use println instead of logInfo to print AM tracking url in YARN client
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89729144
Writing to stdout/stderr defeats the point of a logging framework, no. I think you could argue that some of these other messages aren't vital at log level ("setting up", "preparing", etc.) and turn those down.
Otherwise I'm afraid logging configuration can't be controlled at the level of statements by end users, and others would say these are useful enough / not a problem enough to disable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89965682
(I did mean you could look at some of these other messages and suggest others that can be demoted to debug, as a partial solution. We did this recently with block manager messages. I only meant that this isn't the right way forward as you are creating a sort of additional uncontrollable log level that mixes with program output.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89724217
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by piaozhexiu <gi...@git.apache.org>.
Github user piaozhexiu commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89728738
@srowen I'd like to turn down pretty much every INFO message from YARN client except the AM url. (See below.) As can be seen, none of these is useful for end users except for the AM url. Unfortunately, I can't selectively turn down other messages since they're all in the same package.
How about if I print the tracking url to stdout and leave the INFO log as is? Then, I can turn off INFO in YARN client.
-----
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15/04/05 06:36:29 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
15/04/05 06:36:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/04/05 06:36:29 INFO spark.SparkContext: Running Spark version 1.3.0
15/04/05 06:36:30 INFO spark.SecurityManager: Changing view acls to: cheolsoop
15/04/05 06:36:30 INFO spark.SecurityManager: Changing modify acls to: cheolsoop
15/04/05 06:36:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cheolsoop); users with modify permissions: Set(cheolsoop)
15/04/05 06:36:30 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/04/05 06:36:30 INFO Remoting: Starting remoting
15/04/05 06:36:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@ip-10-99-146-254.ec2.internal:64877]
15/04/05 06:36:30 INFO util.Utils: Successfully started service 'sparkDriver' on port 64877.
15/04/05 06:36:30 INFO spark.SparkEnv: Registering MapOutputTracker
15/04/05 06:36:30 INFO spark.SparkEnv: Registering BlockManagerMaster
15/04/05 06:36:30 INFO storage.DiskBlockManager: Created local directory at /mnt/spark_tmp/spark-e95cf4af-ec65-469a-ad1a-827d1149eeab/blockmgr-ec922fd7-58c5-497a-a952-247fcb3ab779
15/04/05 06:36:30 INFO storage.MemoryStore: MemoryStore started with capacity 265.4 MB
15/04/05 06:36:31 INFO spark.HttpFileServer: HTTP File server directory is /mnt/spark_tmp/spark-84f23ed2-ebf3-4022-93e1-fbb31325ab3f/httpd-128e4efa-c666-4311-b1ee-c0868eaca4bc
15/04/05 06:36:31 INFO spark.HttpServer: Starting HTTP Server
15/04/05 06:36:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/04/05 06:36:31 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:36543
15/04/05 06:36:31 INFO util.Utils: Successfully started service 'HTTP file server' on port 36543.
15/04/05 06:36:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/04/05 06:36:31 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/04/05 06:36:31 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:47936
15/04/05 06:36:31 INFO util.Utils: Successfully started service 'SparkUI' on port 47936.
15/04/05 06:36:31 INFO ui.SparkUI: Started SparkUI at http://ip-10-99-146-254.ec2.internal:47936
15/04/05 06:36:31 INFO client.RMProxy: Connecting to ResourceManager at /10.171.119.231:9022
15/04/05 06:36:31 INFO yarn.Client: Requesting a new application from cluster with 1300 NodeManagers
15/04/05 06:36:31 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (10240 MB per container)
15/04/05 06:36:31 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
15/04/05 06:36:31 INFO yarn.Client: Setting up container launch context for our AM
15/04/05 06:36:31 INFO yarn.Client: Preparing resources for our AM container
15/04/05 06:36:32 INFO yarn.Client: Uploading resource file:/mnt/tmp/bdp-clients/cheolsoop/20150405_063623.027801.prodsparkshell13/jars/spark-1.3.0/lib/spark-assembly-1.3.1-SNAPSHOT-hadoop2.4.0.jar -> hdfs://10.171.119.231:9000/user/cheolsoop/.sparkStaging/application_1426271585556_249126/spark-assembly-1.3.1-SNAPSHOT-hadoop2.4.0.jar
15/04/05 06:36:35 INFO yarn.Client: Setting up the launch environment for our AM container
15/04/05 06:36:35 INFO spark.SecurityManager: Changing view acls to: cheolsoop
15/04/05 06:36:35 INFO spark.SecurityManager: Changing modify acls to: cheolsoop
15/04/05 06:36:35 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cheolsoop); users with modify permissions: Set(cheolsoop)
15/04/05 06:36:35 INFO yarn.Client: Submitting application 249126 to ResourceManager
15/04/05 06:36:35 INFO impl.YarnClientImpl: Submitted application application_1426271585556_249126
15/04/05 06:36:36 INFO yarn.Client: Application report for application_1426271585556_249126 (state: ACCEPTED)
15/04/05 06:36:36 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.adhoc.cheolsoop
start time: 1428215795227
final status: UNDEFINED
tracking URL: http://10.171.119.231:9046/proxy/application_1426271585556_249126/
user: cheolsoop
15/04/05 06:36:37 INFO yarn.Client: Application report for application_1426271585556_249126 (state: ACCEPTED)
-----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by piaozhexiu <gi...@git.apache.org>.
Github user piaozhexiu closed the pull request at:
https://github.com/apache/spark/pull/5362
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by piaozhexiu <gi...@git.apache.org>.
Github user piaozhexiu commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89914799
OK, I guess it's hard to come up with a suggestion that makes everyone happy. I'll make custom changes to my own branch and move on.
Thank you for the discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6712][YARN] Allow lower the log level i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5362#issuecomment-89727695
No, println isn't appropriate here. That removes control over the logging entirely. Instead, what log messages do you find noisy? maybe they can be turned *down* since this message is appropriate at the info level. Or, can you not just selectively disable messages from packages in your log4j config? or is the noise from the same package?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org