You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2014/08/05 03:27:30 UTC
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/1776
[WIP] [SPARK-2655] Change logging level from INFO to DEBUG
Currently, it's too noisy at the default level INFO, so it's better to put the logging about runtime details into DEBUG level.
Basic rule is that, DEBUG is used for developer or debugging, INFO is used for user (who does not know much of the details). So if some loggings do not provide helpful messages for users, it should be put in DEBUG level.
The progress of Job/stage/task is helpful for user, should be keep in INFO level. (It will be better to put all messages of task completeness into one line, not addressed in this PR)
The logging level in third-parties should be set to WARN by default.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark log
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1776.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1776
----
commit 78d08343d35ca17df9c2d1956b4065e35680482b
Author: Davies Liu <da...@gmail.com>
Date: 2014-08-04T18:31:25Z
use WARN as default logging level in log4j
commit 086b55ec608b513555c8c037a17477453560931b
Author: Davies Liu <da...@gmail.com>
Date: 2014-08-04T19:37:13Z
change logging level from INFO to DEBUG
commit d03d6075dca9d73f5aab4c3da267330adf9002ba
Author: Davies Liu <da...@gmail.com>
Date: 2014-08-05T00:06:04Z
change log level to DEBUG
commit ce7f3c2e41d4690fc5b63e517ff24ec3f0eeb4eb
Author: Davies Liu <da...@gmail.com>
Date: 2014-08-05T00:57:24Z
rollback default logging level to INFO
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-54379827
Do you mind closing this for now? Feel free to open a new PR later if you want to continue discussion of log-level cleanup.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919922
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ---
@@ -38,14 +38,14 @@ class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf) extends Log
/** Remove a dead executor from the driver actor. This is only called on the driver side. */
def removeExecutor(execId: String) {
tell(RemoveExecutor(execId))
- logInfo("Removed " + execId + " successfully in removeExecutor")
+ logDebug("Removed " + execId + " successfully in removeExecutor")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51552785
As I said, we do *not* want users to have to change anything when rerunning. It's just an extra step and it makes it much harder to report problems. From my point of view:
- The executor logs can remain at INFO and contain what they have now; you only look at those if you had a problem
- The driver could be less verbose; we can eventually put it on WARN, but for now we should at least reduce their verbosity. For example long-term we might make spark-shell launch with WARN while executors launch with INFO.
Also IMO it's too late to change this stuff for 1.1; this is a pretty big change. We can look at it for 1.2 though, and I'd also be okay removing a few noisy log messages for 1.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919859
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -535,12 +535,12 @@ class DAGScheduler(
* Cancel a job that is running or waiting in the queue.
*/
def cancelJob(jobId: Int) {
- logInfo("Asked to cancel job " + jobId)
+ logDebug("Asked to cancel job " + jobId)
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919803
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -170,8 +170,8 @@ private[spark] class SecurityManager(sparkConf: SparkConf) extends Logging {
}
private[spark] def setViewAcls(defaultUsers: Seq[String], allowedUsers: String) {
- viewAcls = (defaultUsers ++ allowedUsers.split(',')).map(_.trim()).filter(!_.isEmpty).toSet
- logInfo("Changing view acls to: " + viewAcls.mkString(","))
+ viewAcls = (defaultUsers ++ allowedUsers.split(',')).map(_.trim()).filter(!_.isEmpty).toSet
+ logDebug("Changing view acls to: " + viewAcls.mkString(","))
--- End diff --
this too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919909
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -518,12 +518,12 @@ private[spark] class BlockManager(
def get(blockId: BlockId): Option[BlockResult] = {
val local = getLocal(blockId)
if (local.isDefined) {
- logInfo(s"Found block $blockId locally")
+ logDebug(s"Found block $blockId locally")
return local
}
val remote = getRemote(blockId)
if (remote.isDefined) {
- logInfo(s"Found block $blockId remotely")
+ logDebug(s"Found block $blockId remotely")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919916
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -908,7 +908,7 @@ private[spark] class BlockManager(
* Remove all blocks belonging to the given broadcast.
*/
def removeBroadcast(broadcastId: Long, tellMaster: Boolean): Int = {
- logInfo(s"Removing broadcast $broadcastId")
+ logDebug(s"Removing broadcast $broadcastId")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919913
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -856,7 +856,7 @@ private[spark] class BlockManager(
// Drop to disk, if storage level requires
if (level.useDisk && !diskStore.contains(blockId)) {
- logInfo(s"Writing block $blockId to disk")
+ logDebug(s"Writing block $blockId to disk")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919838
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -167,7 +167,7 @@ private[spark] class Executor(
SparkEnv.set(env)
Thread.currentThread.setContextClassLoader(replClassLoader)
val ser = SparkEnv.get.closureSerializer.newInstance()
- logInfo(s"Running $taskName (TID $taskId)")
+ logDebug(s"Running $taskName (TID $taskId)")
--- End diff --
also the following two
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919907
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -518,12 +518,12 @@ private[spark] class BlockManager(
def get(blockId: BlockId): Option[BlockResult] = {
val local = getLocal(blockId)
if (local.isDefined) {
- logInfo(s"Found block $blockId locally")
+ logDebug(s"Found block $blockId locally")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919867
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -859,7 +859,7 @@ class DAGScheduler(
return
}
- logInfo("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd + ")")
+ logDebug("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd + ")")
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919824
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -810,7 +810,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Fetch the file locally in case a job is executed using DAGScheduler.runLocally().
Utils.fetchFile(path, new File(SparkFiles.getRootDirectory()), conf, env.securityManager)
- logInfo("Added file " + path + " at " + key + " with timestamp " + addedFiles(key))
+ logDebug("Added file " + path + " at " + key + " with timestamp " + addedFiles(key))
--- End diff --
I'd keep all the ones in SparkContext in INFO. I think they are kind of important ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919874
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -1117,7 +1117,7 @@ class DAGScheduler(
failJobAndIndependentStages(job, s"Job aborted due to stage failure: $reason")
}
if (dependentJobs.isEmpty) {
- logInfo("Ignoring failure of " + failedStage + " because all jobs depending on it are done")
+ logDebug("Ignoring failure of " + failedStage + " because all jobs depending on it are done")
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919951
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo(
if (storageLevel.useMemory) {
_blocks.put(blockId, BlockStatus(storageLevel, memSize, 0, 0))
_remainingMem -= memSize
- logInfo("Added %s in memory on %s (size: %s, free: %s)".format(
+ logDebug("Added %s in memory on %s (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(memSize),
Utils.bytesToString(_remainingMem)))
}
if (storageLevel.useDisk) {
_blocks.put(blockId, BlockStatus(storageLevel, 0, diskSize, 0))
- logInfo("Added %s on disk on %s (size: %s)".format(
+ logDebug("Added %s on disk on %s (size: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(diskSize)))
}
if (storageLevel.useOffHeap) {
_blocks.put(blockId, BlockStatus(storageLevel, 0, 0, tachyonSize))
- logInfo("Added %s on tachyon on %s (size: %s)".format(
+ logDebug("Added %s on tachyon on %s (size: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919977
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -325,7 +325,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long)
currentMemory += size
}
val valuesOrBytes = if (deserialized) "values" else "bytes"
- logInfo("Block %s stored as %s in memory (estimated size %s, free %s)".format(
+ logDebug("Block %s stored as %s in memory (estimated size %s, free %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919918
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -920,7 +920,7 @@ private[spark] class BlockManager(
* Remove a block from both memory and disk.
*/
def removeBlock(blockId: BlockId, tellMaster: Boolean = true): Unit = {
- logInfo(s"Removing block $blockId")
+ logDebug(s"Removing block $blockId")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919930
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ---
@@ -38,14 +38,14 @@ class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf) extends Log
/** Remove a dead executor from the driver actor. This is only called on the driver side. */
def removeExecutor(execId: String) {
tell(RemoveExecutor(execId))
- logInfo("Removed " + execId + " successfully in removeExecutor")
+ logDebug("Removed " + execId + " successfully in removeExecutor")
}
/** Register the BlockManager's id with the driver. */
def registerBlockManager(blockManagerId: BlockManagerId, maxMemSize: Long, slaveActor: ActorRef) {
- logInfo("Trying to register BlockManager")
+ logDebug("Trying to register BlockManager")
tell(RegisterBlockManager(blockManagerId, maxMemSize, slaveActor))
- logInfo("Registered BlockManager")
+ logDebug("Registered BlockManager")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919910
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -837,7 +837,7 @@ private[spark] class BlockManager(
blockId: BlockId,
data: Either[Array[Any], ByteBuffer]): Option[BlockStatus] = {
- logInfo(s"Dropping block $blockId from memory")
+ logDebug(s"Dropping block $blockId from memory")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919880
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -431,7 +431,7 @@ private[spark] class TaskSetManager(
// a good proxy to task serialization time.
// val timeTaken = clock.getTime() - startTime
val taskName = s"task ${info.id} in stage ${taskSet.id}"
- logInfo("Starting %s (TID %d, %s, %s, %d bytes)".format(
+ logDebug("Starting %s (TID %d, %s, %s, %d bytes)".format(
--- End diff --
this is progress reporting, isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51502324
Keep in mind with these messages that INFO level is also used in executor logs, which are the only way to debug a problem after a job ran. I wouldn't remove too many of those -- asking the user to rerun their job with DEBUG is not something we want to do on a regular basis.
If some of these clutter output on the driver, we can deal with them separately. In my experience the vast majority of messages on the driver are from the job scheduler, and we can just replace that with something that prints a progress bar nicely.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51265949
Currently, it's too noisy by default. We have two options,
1. change to default logging level to WARN, then we need to move some of them from INFO to WARN.
2. change bunch of logs from INFO to DEBUG.
Which one will be reasonable? I think we should have a better default behavior for uses.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51263718
@davies why did you change so many logInfos in the code to logDebug? I'm not sure all of these are developer info. If you want to change the default log level, just leave these as they are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51253325
cc @JoshRosen @mateiz
If this rule make sense, I will continue to change level of others.
I had not figure out a good way to do progressbar in console, we could do that later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919933
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ---
@@ -57,7 +57,7 @@ class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf) extends Log
tachyonSize: Long): Boolean = {
val res = askDriverWithReply[Boolean](
UpdateBlockInfo(blockManagerId, blockId, storageLevel, memSize, diskSize, tachyonSize))
- logInfo("Updated info of block " + blockId)
+ logDebug("Updated info of block " + blockId)
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51266551
I agree with Davies suggestion that DEBUG level should be for Spark
internals developers to turn on when doing debugging an issue, and INFO
level should be for Spark users to see generally what's happening.
On Tue, Aug 5, 2014 at 2:54 PM, Davies Liu <no...@github.com> wrote:
> Currently, it's too noisy by default. We have two options,
>
> 1. change to default logging level to WARN, then we need to move some
> of them from INFO to WARN.
> 2. change bunch of logs from INFO to DEBUG.
>
> Which one will be reasonable? I think we should have a better default
> behavior for uses.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/1776#issuecomment-51265949>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919953
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo(
_blocks.remove(blockId)
if (blockStatus.storageLevel.useMemory) {
_remainingMem += blockStatus.memSize
- logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
+ logDebug("Removed %s on %s in memory (size: %s, free: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919955
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo(
_blocks.remove(blockId)
if (blockStatus.storageLevel.useMemory) {
_remainingMem += blockStatus.memSize
- logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
+ logDebug("Removed %s on %s in memory (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize),
Utils.bytesToString(_remainingMem)))
}
if (blockStatus.storageLevel.useDisk) {
- logInfo("Removed %s on %s on disk (size: %s)".format(
+ logDebug("Removed %s on %s on disk (size: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919873
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -1086,7 +1086,7 @@ class DAGScheduler(
handleJobCancellation(jobId, s"because Stage $stageId was cancelled")
}
case None =>
- logInfo("No active jobs to kill for Stage " + stageId)
+ logDebug("No active jobs to kill for Stage " + stageId)
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919794
--- Diff: core/src/main/scala/org/apache/spark/HttpServer.scala ---
@@ -50,7 +50,7 @@ private[spark] class HttpServer(resourceBase: File, securityManager: SecurityMan
if (server != null) {
throw new ServerStateException("Server is already started")
} else {
- logInfo("Starting HTTP Server")
+ logDebug("Starting HTTP Server")
--- End diff --
This logging message is useless to me... maybe we can remove it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919966
--- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -158,7 +158,7 @@ private[spark] class DiskBlockManager(shuffleBlockManager: ShuffleBlockManager,
" Ignoring this directory.")
None
} else {
- logInfo(s"Created local directory at $localDir")
+ logDebug(s"Created local directory at $localDir")
--- End diff --
i'd leave this as info
this is very important information when configuration is wrong
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919949
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo(
if (storageLevel.useMemory) {
_blocks.put(blockId, BlockStatus(storageLevel, memSize, 0, 0))
_remainingMem -= memSize
- logInfo("Added %s in memory on %s (size: %s, free: %s)".format(
+ logDebug("Added %s in memory on %s (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(memSize),
Utils.bytesToString(_remainingMem)))
}
if (storageLevel.useDisk) {
_blocks.put(blockId, BlockStatus(storageLevel, 0, diskSize, 0))
- logInfo("Added %s on disk on %s (size: %s)".format(
+ logDebug("Added %s on disk on %s (size: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919860
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
@@ -535,12 +535,12 @@ class DAGScheduler(
* Cancel a job that is running or waiting in the queue.
*/
def cancelJob(jobId: Int) {
- logInfo("Asked to cancel job " + jobId)
+ logDebug("Asked to cancel job " + jobId)
eventProcessActor ! JobCancelled(jobId)
}
def cancelJobGroup(groupId: String) {
- logInfo("Asked to cancel job group " + groupId)
+ logDebug("Asked to cancel job group " + groupId)
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919835
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -167,7 +167,7 @@ private[spark] class Executor(
SparkEnv.set(env)
Thread.currentThread.setContextClassLoader(replClassLoader)
val ser = SparkEnv.get.closureSerializer.newInstance()
- logInfo(s"Running $taskName (TID $taskId)")
+ logDebug(s"Running $taskName (TID $taskId)")
--- End diff --
let's keep this on. this is only chatty if you run it in single node. in distributed mode, it is very important information to log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919852
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -188,7 +188,7 @@ class HadoopRDD[K, V](
val iter = new NextIterator[(K, V)] {
val split = theSplit.asInstanceOf[HadoopPartition]
- logInfo("Input split: " + split.inputSplit)
+ logDebug("Input split: " + split.inputSplit)
--- End diff --
i'd leave this one as info too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51139737
QA tests have started for PR 1776. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17904/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51435412
I went through all the changes and made comments on ones I think that should be kept INFO. A lot of them are too chatty, but some of them are useful to keep based on my experience troubleshooting, especially for new users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51139805
QA results for PR 1776:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17904/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919959
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo(
_blocks.remove(blockId)
if (blockStatus.storageLevel.useMemory) {
_remainingMem += blockStatus.memSize
- logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
+ logDebug("Removed %s on %s in memory (size: %s, free: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize),
Utils.bytesToString(_remainingMem)))
}
if (blockStatus.storageLevel.useDisk) {
- logInfo("Removed %s on %s on disk (size: %s)".format(
+ logDebug("Removed %s on %s on disk (size: %s)".format(
blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.diskSize)))
}
if (blockStatus.storageLevel.useOffHeap) {
- logInfo("Removed %s on %s on tachyon (size: %s)".format(
+ logDebug("Removed %s on %s on tachyon (size: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919929
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ---
@@ -38,14 +38,14 @@ class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf) extends Log
/** Remove a dead executor from the driver actor. This is only called on the driver side. */
def removeExecutor(execId: String) {
tell(RemoveExecutor(execId))
- logInfo("Removed " + execId + " successfully in removeExecutor")
+ logDebug("Removed " + execId + " successfully in removeExecutor")
}
/** Register the BlockManager's id with the driver. */
def registerBlockManager(blockManagerId: BlockManagerId, maxMemSize: Long, slaveActor: ActorRef) {
- logInfo("Trying to register BlockManager")
+ logDebug("Trying to register BlockManager")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919807
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -179,8 +179,8 @@ private[spark] class SecurityManager(sparkConf: SparkConf) extends Logging {
}
private[spark] def setUIAcls(aclSetting: Boolean) {
- uiAclsOn = aclSetting
- logInfo("Changing acls enabled to: " + uiAclsOn)
+ uiAclsOn = aclSetting
+ logDebug("Changing acls enabled to: " + uiAclsOn)
--- End diff --
and this one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919915
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ---
@@ -898,7 +898,7 @@ private[spark] class BlockManager(
*/
def removeRdd(rddId: Int): Int = {
// TODO: Avoid a linear scan by creating another mapping of RDD.id to blocks.
- logInfo(s"Removing RDD $rddId")
+ logDebug(s"Removing RDD $rddId")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919946
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -339,7 +339,7 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus
blockManagerIdByExecutor(id.executorId) = id
}
- logInfo("Registering block manager %s with %s RAM".format(
+ logDebug("Registering block manager %s with %s RAM".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919942
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -69,7 +69,7 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus
def receive = {
case RegisterBlockManager(blockManagerId, maxMemSize, slaveActor) =>
- logInfo("received a register")
+ logDebug("received a register")
--- End diff --
i'd remove this and maybe just keep a general logDebug to log all the messages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919947
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo(
if (storageLevel.useMemory) {
_blocks.put(blockId, BlockStatus(storageLevel, memSize, 0, 0))
_remainingMem -= memSize
- logInfo("Added %s in memory on %s (size: %s, free: %s)".format(
+ logDebug("Added %s in memory on %s (size: %s, free: %s)".format(
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919888
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -503,7 +503,7 @@ private[spark] class TaskSetManager(
isZombie = true
}
} else {
- logInfo("Ignoring task-finished event for " + info.id + " in stage " + taskSet.id +
+ logDebug("Ignoring task-finished event for " + info.id + " in stage " + taskSet.id +
--- End diff --
i would make this info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919800
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -146,7 +146,7 @@ private[spark] class SecurityManager(sparkConf: SparkConf) extends Logging {
setViewAcls(defaultAclUsers, sparkConf.get("spark.ui.view.acls", ""))
private val secretKey = generateSecretKey()
- logInfo("SecurityManager: authentication " + (if (authOn) "enabled" else "disabled") +
+ logDebug("SecurityManager: authentication " + (if (authOn) "enabled" else "disabled") +
--- End diff --
This would be very useful. I'd keep it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919971
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -176,7 +176,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long)
val entry = entries.remove(blockId)
if (entry != null) {
currentMemory -= entry.size
- logInfo(s"Block $blockId of size ${entry.size} dropped from memory (free $freeMemory)")
+ logDebug(s"Block $blockId of size ${entry.size} dropped from memory (free $freeMemory)")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919943
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala ---
@@ -225,7 +225,7 @@ class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus
}
private def removeExecutor(execId: String) {
- logInfo("Trying to remove executor " + execId + " from BlockManagerMaster.")
+ logDebug("Trying to remove executor " + execId + " from BlockManagerMaster.")
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51543727
Before moving on, maybe the first question is that, does Spark 1.1 is so stable that we can trust it work as expected? If yes in most cases, I think reduce the chatty loggings will make sense. If not, then we should continue with that and revisit in the future.
If user has some problems, she will try to ru-run the program several times to make sure that the problem is re-producable. During these, turning on DEBUG loggings will be reasonable. It will more helpful if she could switch the logging on/off by single argument, such as "-v" or "-q".
In current code base, most of the loggings happens in normal code path, they show what exactly happen right now. They are useful for developer to troubleshoot, but just be noisy to user. User should be focus on her business login, not the details of spark.
We will not remove these loggings, just want to hidden them to users by default, only keep the really useful loggings, such as address of web UI, and progress loggings (replaces by other cool stuff will be better).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1776#discussion_r15919967
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -56,7 +56,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long)
(maxMemory * unrollFraction).toLong
}
- logInfo("MemoryStore started with capacity %s".format(Utils.bytesToString(maxMemory)))
+ logDebug("MemoryStore started with capacity %s".format(Utils.bytesToString(maxMemory)))
--- End diff --
i'd leave this as info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by davies <gi...@git.apache.org>.
Github user davies closed the pull request at:
https://github.com/apache/spark/pull/1776
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1776#issuecomment-51552928
BTW the problem is not "does it work as expected in most cases", it's "is it easy to debug when it's not working as expected". We need to make it as easy as possible for users to diagnose problems and report them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org