You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2024/05/23 05:26:00 UTC

[jira] [Resolved] (HDDS-10898) Why the log4j.properties configuration file is mandatory?

     [ https://issues.apache.org/jira/browse/HDDS-10898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Doroszlai resolved HDDS-10898.
-------------------------------------
    Fix Version/s: 1.5.0
       Resolution: Fixed

> Why the log4j.properties configuration file is mandatory?
> ---------------------------------------------------------
>
>                 Key: HDDS-10898
>                 URL: https://issues.apache.org/jira/browse/HDDS-10898
>             Project: Apache Ozone
>          Issue Type: Improvement
>            Reporter: Vyacheslav Tutrinov
>            Assignee: Vyacheslav Tutrinov
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.5.0
>
>
> h1. Imagine a case:
> 1. I know what the Ozone is and I'm familiar with its basic environment variables (e.g. *OZONE_CONF_DIR*, *OZONE_OPTS*)
> 2. There is the *ozone-site.xml* (empty, so far) configuration file.
> 3. The conf file is deployed to a specific conf dir on my infrastructure's hosts
> 4. Ozone binaries are deployed too
> 5. Try to start an ozone's service instance (e.g. *scm*):
> Cmd:
> {code}
> export OZONE_CONF_DIR=/my/custom/path/to/ozone/configurations/files
> ${OZONE_HOME}/bin/ozone scm --init
> ${OZONE_HOME}/bin/ozone scm
> {code}
> Output:
> {code:}
> # a lot of info about classpath + default conf properties
> 2024-05-22 07:30:26,012 [main] ERROR server.StorageContainerManagerStarter: SCM start failed with exception
> java.lang.IllegalArgumentException: ozone.metadata.dirs must be defined.
> 	at org.apache.hadoop.hdds.server.ServerUtils.getOzoneMetaDirPath(ServerUtils.java:284)
> 	at org.apache.hadoop.hdds.server.ServerUtils.getScmDbDir(ServerUtils.java:158)
> 	at org.apache.hadoop.hdds.scm.server.SCMStorageConfig.<init>(SCMStorageConfig.java:49)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:362)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:611)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:623)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:171)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:145)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:74)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:48)
> 	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
> 	at picocli.CommandLine.access$1500(CommandLine.java:148)
> 	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
> 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
> 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
> 	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
> 	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
> 	at picocli.CommandLine.execute(CommandLine.java:2170)
> 	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
> 	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:63)
> ozone.metadata.dirs must be defined.
> {code}
> 6. OK, Mr. Ozone, let me define the config property
> {code:xml}
> <!-- cat /my/custom/path/to/ozone/configurations/files/ozone-site.xml -->
> <configuration>
> 	<property>
> 		<name>ozone.metadata.dirs</name>
> 		<value>/data/metadata</value>
> 	</property>
> </configuration>
> {code}
> 7. Another attempt to start the SCM:
> Cmd:
> {code}
> ${OZONE_HOME}/bin/ozone scm
> {code}
> Output:
> {code}
> # Hmm, the same output ...
> 2024-05-22 08:01:23,365 [main] ERROR server.StorageContainerManagerStarter: SCM start failed with exception
> java.lang.IllegalArgumentException: ozone.metadata.dirs must be defined.
> 	at org.apache.hadoop.hdds.server.ServerUtils.getOzoneMetaDirPath(ServerUtils.java:284)
> 	at org.apache.hadoop.hdds.server.ServerUtils.getScmDbDir(ServerUtils.java:158)
> 	at org.apache.hadoop.hdds.scm.server.SCMStorageConfig.<init>(SCMStorageConfig.java:49)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.<init>(StorageContainerManager.java:362)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:611)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:623)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:171)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:145)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:74)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:48)
> 	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
> 	at picocli.CommandLine.access$1500(CommandLine.java:148)
> 	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
> 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
> 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
> 	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
> 	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
> 	at picocli.CommandLine.execute(CommandLine.java:2170)
> 	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
> 	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
> 	at org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.main(StorageContainerManagerStarter.java:63)
> ozone.metadata.dirs must be defined.
> {code}
> h2. What's going on?
> Provided *OZONE_CONF_DIR* doesn't affect the ozone behavior, but why? Let's dive into bin/ozone shell script.
> It has the following lines:
> {code}
> ozone_bootstrap
> . "${OZONE_LIBEXEC_DIR}/ozone-config.sh"
> {code}
> The ozone-config.sh has the following lines:
> {code}
> #
> # Setup the base-line environment
> #
> ozone_find_confdir
> {code}
> And the ozone-functions.sh related code is:
> {code}
> ## @description  Locate Ozone's configuration directory
> ## @audience     private
> ## @stability    evolving
> ## @replaceable  no
> function ozone_find_confdir
> {
>   ozone_deprecate_envvar HADOOP_CONF_DIR OZONE_CONF_DIR
>   local conf_dir=etc/hadoop
>   if [[ -n "${OZONE_CONF_DIR}" ]] && ozone_verify_confdir "${OZONE_CONF_DIR}"; then
>     : # OK
>   elif [[ -n "${OZONE_HOME}" ]] && ozone_verify_confdir "${OZONE_HOME}/${conf_dir}"; then
>     OZONE_CONF_DIR="${OZONE_HOME}/${conf_dir}"
>   elif [[ -n "${OZONE_LIBEXEC_DIR}" ]] && ozone_verify_confdir "${OZONE_LIBEXEC_DIR}/../${conf_dir}"; then
>     OZONE_CONF_DIR=$(ozone_abs "${OZONE_LIBEXEC_DIR}/../${conf_dir}")
>   else
>     OZONE_CONF_DIR="${OZONE_HOME}/${conf_dir}" # not verified yet
>     ozone_error "WARNING: OZONE_CONF_DIR not defined and cannot be found, setting in OZONE_HOME: ${OZONE_CONF_DIR}."
>   fi
>   export OZONE_CONF_DIR
>   ozone_using_envvar OZONE_CONF_DIR
>   ozone_set_deprecated_var HADOOP_CONF_DIR OZONE_CONF_DIR
> }
> ## @description  Validate ${OZONE_CONF_DIR}
> ## @audience     public
> ## @stability    stable
> ## @replaceable  yes
> ## @return       will exit on failure conditions
> function ozone_verify_confdir
> {
>   # Check only log4j.properties by default.
>   # --loglevel does not work without logger settings in log4j.properties.
>   [[ -f "${1:?ozone_verify_confir requires parameter}/log4j.properties" ]]
> }
> {code}
> The last code block shows us that the ${OZONE_CONF_DIR}/log4j.properties configuration file is mandatory and if it's not presented in $OZONE_CONF_DIR the conf dir will be overridden to ${OZONE_HOME}/etc/hadoop.
> However the ozone services can work without the log4.properties configuration file (we won't see any logs, but the service will work anyway). Moreover, the ozone-site.xml looks more applicable for the conf dir validation purpose above (at least SCM can't start without some conf properties inside)
> h3. Suggestion
> Replace the log4j.properties with ozone-site.xml in *ozone_verify_confdir* function in ${OZONE_LIBEXEC_DIR}/ozone-functions.sh script



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org