You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2020/09/09 09:18:00 UTC
[jira] [Comment Edited] (TEZ-4039) Tez should inject dag id, query
id into MDC
[ https://issues.apache.org/jira/browse/TEZ-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192271#comment-17192271 ]
László Bodor edited comment on TEZ-4039 at 9/9/20, 9:17 AM:
------------------------------------------------------------
there is a working version in: [^TEZ-4039.05.patch]
testing:
{code}
wget -qO- "https://issues.apache.org/jira/secure/attachment/13011227/TEZ-4039.05.patch" | git apply -3 -p0
mvn clean install -pl tez-api -pl tez-dag -pl tez-common -pl tez-runtime-internals -pl tez-tests -pl tez-examples -Dtest=TestTezJobs#testHashJoinExampleWithLogPattern
cd tez-tests/target/org.apache.tez.test.TestTezJobs/org.apache.tez.test.TestTezJobs-logDir-nm-0_0
# check syslog_dag* and syslog_attempt* files
{code}
[~jeagles], [~prasanth_j]: could you please take a look?
[^TEZ-4039.05.patch] is a working version, I've included manual testing steps
the change includes all features discussed above + a configurable layout pattern (which turned out to be a very convenient way, that can replace/eliminate painful enviroment-dependent steps, etc. deploying log4j configuration files), which is turned off by default
was (Author: abstractdog):
there is a working version in: [^TEZ-4039.04.patch]
testing:
{code}
wget -qO- "https://issues.apache.org/jira/secure/attachment/13011227/TEZ-4039.05.patch" | git apply -3 -p0
mvn clean install -pl tez-api -pl tez-dag -pl tez-common -pl tez-runtime-internals -pl tez-tests -pl tez-examples -Dtest=TestTezJobs#testHashJoinExampleWithLogPattern
cd tez-tests/target/org.apache.tez.test.TestTezJobs/org.apache.tez.test.TestTezJobs-logDir-nm-0_0
# check syslog_dag* and syslog_attempt* files
{code}
> Tez should inject dag id, query id into MDC
> -------------------------------------------
>
> Key: TEZ-4039
> URL: https://issues.apache.org/jira/browse/TEZ-4039
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Prasanth Jayachandran
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4039.01.patch, TEZ-4039.02.patch, TEZ-4039.03.patch, TEZ-4039.04.patch, TEZ-4039.05.patch
>
>
> Tez currently uses CallableWithNdc to store thread specific context. It should also inject the context into MDC so that pattern layout can dump the contexts from MDC (with NDC it is not possible to read the context in pattern lyaout).
> Hive for example, sets queryId in the MDC and pattern layout prints the queryId
>
> {code:java}
> %d{ISO8601} %-5p [%t (%X{queryId})] %c{2}: %m%n
> {code}
> Llap sets dagId, fragmentId and queryId into MDC which is used for queryId based routing of logging.
> Similarly, Tez AM should set dagId and queryId (if available) into MDC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)