You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/09/24 00:23:00 UTC
[jira] [Commented] (HDFS-17570) Respect Non-Default HADOOP_ROOT_LOGGER when HADOOP_DAEMON_ROOT_LOGGER is not specified in Daemon mode
[ https://issues.apache.org/jira/browse/HDFS-17570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18022292#comment-18022292 ]
ASF GitHub Bot commented on HDFS-17570:
---------------------------------------
github-actions[bot] commented on PR #6922:
URL: https://github.com/apache/hadoop/pull/6922#issuecomment-3325980537
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
> Respect Non-Default HADOOP_ROOT_LOGGER when HADOOP_DAEMON_ROOT_LOGGER is not specified in Daemon mode
> -----------------------------------------------------------------------------------------------------
>
> Key: HDFS-17570
> URL: https://issues.apache.org/jira/browse/HDFS-17570
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: wuchang
> Priority: Major
> Labels: pull-request-available
>
> h1. Description of PR
> the {{-Dhadoop.root.logger}} will use the value of {{{}HADOOP_ROOT_LOGGER{}}}.
> In most cases, hadoop administrator tends to setup {{HADOOP_ROOT_LOGGER}} directly, it is effective in non-daemon mode and most hadoop administrator are not even aware of the existence of {{{}HADOOP_DAEMON_ROOT_LOGGER{}}}.
> So when he started up HADOOP as daemon mode, no matter how he set up {{{}HADOOP_ROOT_LOGGER{}}}, the {{-Dhadoop.root.logger}} is not changed because in Daemon mode, the {{HADOOP_ROOT_LOGGER}} is set up by {{HADOOP_DAEMON_ROOT_LOGGER}} which is in fact a default value. The code is as below:
>
> {code:java}
> if [[ "${HADOOP_DAEMON_MODE}" != "default" ]]; then
> HADOOP_ROOT_LOGGER="${HADOOP_DAEMON_ROOT_LOGGER}"
> if [[ "${HADOOP_SUBCMD_SECURESERVICE}" = true ]]; then
> HADOOP_LOGFILE="hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.log"
> else
> HADOOP_LOGFILE="hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.log"
> fi
> fi{code}
>
> This makes log setup and troubleshooting becomes very time consuming before we finally figure out the reason and the best practice without reading the code.
> h1. My change is:
> In daemon mode
> * If user has customize(changed) {{HADOOP_ROOT_LOGGER}} but no {{{}HADOOP_DAEMON_ROOT_LOGGER{}}}, then we will respect {{HADOOP_ROOT_LOGGER}} as the final {{-Dhadoop.root.logger}} value, instead of setting up {{HADOOP_ROOT_LOGGER}} with the default {{HADOOP_DAEMON_ROOT_LOGGER}} and print a warning.
> * In other cases, we will use {{{}HADOOP_DAEMON_ROOT_LOGGER{}}}(no matter it is default or customized), we will setup the {{HADOOP_ROOT_LOGGER}} with this customized {{HADOOP_DAEMON_ROOT_LOGGER}}
> h3. How was this patch tested?
> ||Testing Case||Expected Result||
> |Customized {{HADOOP_LOGLEVEL}}|Respect {{HADOOP_LOGLEVEL}}|
> |Without customizing anything|Respect default {{HADOOP_DAEMON_ROOT_LOGGER}} in daemon mode|
> |Customize the {{HADOOP_DAEMON_ROOT_LOGGER}} without customizing {{HADOOP_ROOT_LOGGER}}|Respect customized {{HADOOP_DAEMON_ROOT_LOGGER}}|
> |Customize the {{HADOOP_ROOT_LOGGER}} but not customize {{HADOOP_DAEMON_ROOT_LOGGER}}|Respect customized {{HADOOP_ROOT_LOGGER}} with a warning|
> |Customize both {{HADOOP_ROOT_LOGGER}} and {{HADOOP_DAEMON_ROOT_LOGGER}}|Respect customized {{HADOOP_DAEMON_ROOT_LOGGER}}|
>
> 1. Customize the {{{}HADOOP_LOGLEVEL{}}}, should respect the {{HADOOP_LOGLEVEL}} and {{HADOOP_DAEMON_ROOT_LOGGER}}
> export HADOOP_LOGLEVEL=DEBUG
>
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# ps -ef|grep NameNode
> hadoop 2852464 1 45 04:04 pts/1 00:00:09 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dlog4j.configuration=file:/home/hadoop/current-hadoop/etc/hadoop/log4j-namenode.properties -XX:+UseG1GC -Xmx4096m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1029 -Dhadoop.security.logger=ERROR,RFAS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dyarn.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dyarn.log.file=test-host-name.dev.corp.com.log -Dyarn.home.dir=/home/hadoop/hadoop-3.2.4 -Dyarn.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-3.2.4/lib/native -Dhadoop.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dhadoop.log.file=test-host-name.dev.corp.com.log -Dhadoop.home.dir=/home/hadoop/hadoop-3.2.4 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=DEBUG,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode{code}
> {{ }}
>
> 2. Without customizing anything, should respect default {{HADOOP_DAEMON_ROOT_LOGGER}}
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# ps -ef|grep NameNode
> hadoop 2855220 1 43 04:05 pts/1 00:00:07 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dlog4j.configuration=file:/home/hadoop/current-hadoop/etc/hadoop/log4j-namenode.properties -XX:+UseG1GC -Xmx4096m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1029 -Dhadoop.security.logger=ERROR,RFAS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dyarn.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dyarn.log.file=test-host-name.dev.corp.com.log -Dyarn.home.dir=/home/hadoop/hadoop-3.2.4 -Dyarn.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-3.2.4/lib/native -Dhadoop.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dhadoop.log.file=test-host-name.dev.corp.com.log -Dhadoop.home.dir=/home/hadoop/hadoop-3.2.4 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode{code}
>
> 3. Customize the {{HADOOP_DAEMON_ROOT_LOGGER}} without customizing {{{}HADOOP_ROOT_LOGGER{}}}, should respect customized {{HADOOP_DAEMON_ROOT_LOGGER}}
> {code:java}
> export HADOOP_DAEMON_ROOT_LOGGER=TRACE,RFA
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# ps -ef|grep NameNode
> hadoop 2857093 1 99 04:06 pts/1 00:00:07 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dlog4j.configuration=file:/home/hadoop/current-hadoop/etc/hadoop/log4j-namenode.properties -XX:+UseG1GC -Xmx4096m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1029 -Dhadoop.security.logger=ERROR,RFAS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dyarn.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dyarn.log.file=test-host-name.dev.corp.com.log -Dyarn.home.dir=/home/hadoop/hadoop-3.2.4 -Dyarn.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-3.2.4/lib/native -Dhadoop.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dhadoop.log.file=test-host-name.dev.corp.com.log -Dhadoop.home.dir=/home/hadoop/hadoop-3.2.4 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=TRACE,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode{code}
>
> 4. Customize the {{HADOOP_ROOT_LOGGER}} but not customize {{{}HADOOP_DAEMON_ROOT_LOGGER{}}}, should respect customized {{HADOOP_ROOT_LOGGER}} with a warning
> {code:java}
> export HADOOP_ROOT_LOGGER=TRACE,console{code}
>
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# sudo -u hadoop bin/hdfs --daemon stop namenode;Using customized HADOOP_ROOT_LOGGER as final hadoop.root.logger. Recommending using HADOOP_LOGLEVEL or customize HADOOP_DAEMON_ROOT_LOGGER for logging in daemon mode.{code}
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# ps -ef|grep NameNode
> hadoop 2874729 1 99 04:16 pts/1 00:00:07 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dlog4j.configuration=file:/home/hadoop/current-hadoop/etc/hadoop/log4j-namenode.properties -XX:+UseG1GC -Xmx4096m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1029 -Dhadoop.security.logger=ERROR,RFAS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dyarn.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dyarn.log.file=test-host-name.dev.corp.com.log -Dyarn.home.dir=/home/hadoop/hadoop-3.2.4 -Dyarn.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-3.2.4/lib/native -Dhadoop.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dhadoop.log.file=test-host-name.dev.corp.com.log -Dhadoop.home.dir=/home/hadoop/hadoop-3.2.4 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=TRACE,console -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode{code}
>
> 5. Customize both {{HADOOP_ROOT_LOGGER}} and {{{}HADOOP_DAEMON_ROOT_LOGGER{}}}, should respect customized {{HADOOP_DAEMON_ROOT_LOGGER}}
> {code:java}
> export HADOOP_DAEMON_ROOT_LOGGER=TRACE,RFA
> export HADOOP_ROOT_LOGGER=TRACE,console{code}
> {code:java}
> root@rccd101-6a:/home/hadoop/current-hadoop# ps -ef|grep NameNode
> hadoop 2877521 1 61 04:18 pts/1 00:00:07 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dlog4j.configuration=file:/home/hadoop/current-hadoop/etc/hadoop/log4j-namenode.properties -XX:+UseG1GC -Xmx4096m -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1029 -Dhadoop.security.logger=ERROR,RFAS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Dyarn.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dyarn.log.file=test-host-name.dev.corp.com.log -Dyarn.home.dir=/home/hadoop/hadoop-3.2.4 -Dyarn.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-3.2.4/lib/native -Dhadoop.log.dir=/home/hadoop/hadoop-3.2.4/../logs -Dhadoop.log.file=test-host-name.dev.corp.com.log -Dhadoop.home.dir=/home/hadoop/hadoop-3.2.4 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=TRACE,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode{code}
>
> h3. For code changes:
> * Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
> * Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
> * If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0|http://www.apache.org/legal/resolved.html#category-a]?
> * If applicable, have you updated the {{{}LICENSE{}}}, {{{}LICENSE-binary{}}}, {{NOTICE-binary}} files?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org