You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jim Brennan (Jira)" <ji...@apache.org> on 2021/02/08 17:21:00 UTC
[jira] [Created] (MAPREDUCE-7319) Log list of mappers at trace
level in ShuffleHandler audit log
Jim Brennan created MAPREDUCE-7319:
--------------------------------------
Summary: Log list of mappers at trace level in ShuffleHandler audit log
Key: MAPREDUCE-7319
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: yarn
Affects Versions: 3.4.0
Reporter: Jim Brennan
Assignee: Jim Brennan
[MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which is logged at DEBUG level. After enabling it, we found that the list of mappers for large jobs was filling up our audit logs. It would be good to move the list of mappers to TRACE level to reduce the logging impact without disabling the log message entirely.
For example a log message like this:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers: [attempt_1512479762132_1318600_1_00_004852_0_10003,
attempt_1512479762132_1318600_1_00_004190_0_10003, attempt_1512479762132_1318600_1_00_004393_0_10003, attempt_1512479762132_1318600_1_00_005057_0_10003, attempt_1512479762132_1318600_1_00_004855_0_10002,
attempt_1512479762132_1318600_1_00_003976_0_10003, attempt_1512479762132_1318600_1_00_004058_0_10003, attempt_1512479762132_1318600_1_00_004355_0_10003, attempt_1512479762132_1318600_1_00_004436_0_10002,
attempt_1512479762132_1318600_1_00_004854_0_10003, attempt_1512479762132_1318600_1_00_005174_0_10004, attempt_1512479762132_1318600_1_00_003972_0_10002, attempt_1512479762132_1318600_1_00_004853_0_10002,
attempt_1512479762132_1318600_1_00_004856_0_10002]
{noformat}
Would become this with {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle for job_1512479762132_1318600 reducer 241 length 482072
{noformat}
And this with {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle for job_1512479762132_1318600 reducer 241 length 482072
2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: shuffle for job_1512479762132_1318600 mappers: [attempt_1512479762132_1318600_1_00_004852_0_10003,
attempt_1512479762132_1318600_1_00_004190_0_10003, attempt_1512479762132_1318600_1_00_004393_0_10003, attempt_1512479762132_1318600_1_00_005057_0_10003, attempt_1512479762132_1318600_1_00_004855_0_10002,
attempt_1512479762132_1318600_1_00_003976_0_10003, attempt_1512479762132_1318600_1_00_004058_0_10003, attempt_1512479762132_1318600_1_00_004355_0_10003, attempt_1512479762132_1318600_1_00_004436_0_10002,
attempt_1512479762132_1318600_1_00_004854_0_10003, attempt_1512479762132_1318600_1_00_005174_0_10004, attempt_1512479762132_1318600_1_00_003972_0_10002, attempt_1512479762132_1318600_1_00_004853_0_10002,
attempt_1512479762132_1318600_1_00_004856_0_10002]
{noformat}
One question is whether there are any downstream consumers of this audit log that might have a problem with this change?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org