[jira] [Commented] (TEZ-2774) Reduce logging in the AM

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730019#comment-14730019 ] 

Hitesh Shah commented on TEZ-2774:
----------------------------------

Comments: 

{code}
LOG.info("Comparing client version with AM version"		
        + ", clientVersion=" + clientVersion		
        + ", AMVersion=" + dagVersionInfo.getVersion());
{code}
  - this has been useful at times when debugging rolling upgrade goof-ups. One-time line per AM instance so would be useful to retain. 

{code}
if (LOG.isDebugEnabled()) {
1089	          LOG.debug("remoteTaskSpec:" + remoteTaskSpec);
1090	        }
{code}
   - Does this need any additional context on what is being logged? i.e. taskAttemptId or taskId? 

{code}
    } else if (vertex.completedTaskCount %50 == 0 || vertex.completedTaskCount == vertex.tasks.size()) {
2082	      LOG.info("Checking tasks for vertex completion for "
2083	          + vertex.logIdentifier
{code}
   - should this be "modulo max(50, numTasks/50)"  i.e log this at max 50 times?

{code}
for (ContainerStatus status : statuses) {
412	          LOG.debug("Container " + status.getContainerId() + " is completed");
413	        }
{code}
  - should enhance to log container exit code, diagnostics etc. 
  - maybe log failed containers at info ( not only in this case but others that were converted to debug )? 
  - not required if AM is shutting down - which is the case if stopStarted is true

{code}
LOG.debug("No taskRequests. Container's idle timeout delay expired or is new. " +
640	              "Releasing container"
641	              + ", containerId=" + heldContainer.getContainer().getId()
642	              + ", containerExpiryTime="
643	              + heldContainer.getContainerExpiryTime()
644	              + ", idleTimeout=" + idleContainerTimeoutMin
645	              + ", taskRequestsCount=" + taskRequests.size()
646	              + ", heldContainers=" + heldContainers.size()
647	              + ", delayedContainers=" + delayedContainerManager.delayedContainers.size()
648	              + ", isNew=" + isNew);
649	        }
639	          releaseUnassignedContainers(	650	
{code}
  - might be worth retaining if this is not a very frequent log. If frequent, then yes we can remove. 

{code}
 if (LOG.isDebugEnabled()) {
810	                LOG.debug("Releasing held container as either there are pending but "
811	                    + " unmatched requests or this is not a session"
812	                    + ", containerId=" + heldContainer.container.getId()
813	                    + ", pendingTasks=" + taskRequests.size()
814	                    + ", isSession=" + getContext().isSession()
815	                    + ". isNew=" + isNew);
816	              }
{code}
   - might be worth retaining with a less heavy message. 

typo: timelineAclsClas - change to TimelineACLManagerClass 

{code}
if (isEnabled) {
82	      allocatorClassName = conf.get(TezConfiguration.TEZ_TASK_SCALE_MEMORY_ALLOCATOR_CLASS,
83	          TezConfiguration.TEZ_TASK_SCALE_MEMORY_ALLOCATOR_CLASS_DEFAULT);
84	    } else {
85	      allocatorClassName = null;
86	    }
{code} 
   - non logging change? 

{code}
 private final long LOG_INTERVAL = 30000l;
{code}
   - from 2 seconds to 30 seconds?

Tez runtime log changes are present. Maybe change jira title? 





















> Reduce logging in the AM
> ------------------------
>
>                 Key: TEZ-2774
>                 URL: https://issues.apache.org/jira/browse/TEZ-2774
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>         Attachments: TEZ-2774.1.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)