[jira] [Commented] (TEZ-3359) Add granular log levels for HistoryLoggingService.

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388454#comment-15388454 ] 

Hitesh Shah commented on TEZ-3359:
----------------------------------

Comments: 

   - ConcurrentHashMap instead of HashMap<TezDAGID, HistoryLogLevel>?
   
{code}
    HistoryEvent historyEvent = event.getHistoryEvent();
142	    if (historyEvent.getEventType() == HistoryEventType.DAG_SUBMITTED) {
143	      dagLogLevel = ((DAGSubmittedEvent)historyEvent).getConf().getEnum(
144	          TezConfiguration.TEZ_HISTORY_LOGLEVEL, amHistoryLogLevel);
145	      dagIdToLogLevel.put(dagId, dagLogLevel);
146	    } else if (historyEvent.getEventType() == HistoryEventType.DAG_RECOVERED) {
147	      if (context.getCurrentDAG() != null) {
148	        dagLogLevel = context.getCurrentDAG().getConf().getEnum(
149	            TezConfiguration.TEZ_HISTORY_LOGLEVEL, amHistoryLogLevel);
150	        dagIdToLogLevel.put(dagId, dagLogLevel);
151	      }
{code}
   - should we change this to a bit more robust code i.e. allow for case insensitive matches. log a warning if there is an invalid which does not match any valid enum or maybe even error out on the client side if either the AM or DAG config setting is invalid? A client side error might be better instead of silently falling back to logging everything which might prove dangerous. 

Patch looks mostly good. 

Minor nit on the tests: I dont believe we are testing a single session with mulitple dags where the AM log level and the 2 dags are all using different log levels. Possibly one of the tests could be enhanced to cover this complex case? 




> Add granular log levels for HistoryLoggingService.
> --------------------------------------------------
>
>                 Key: TEZ-3359
>                 URL: https://issues.apache.org/jira/browse/TEZ-3359
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Harish Jaiprakash
>            Assignee: Harish Jaiprakash
>         Attachments: TEZ-3359.01.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS. Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs, or all logs and so on.
> * Disable logging counters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)