You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Arpit Agarwal (Jira)" <ji...@apache.org> on 2023/05/31 13:52:00 UTC
[jira] [Reopened] (HDDS-7002) Unnable to deploy ozone in cluster mode
[ https://issues.apache.org/jira/browse/HDDS-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arpit Agarwal reopened HDDS-7002:
---------------------------------
> Unnable to deploy ozone in cluster mode
> ---------------------------------------
>
> Key: HDDS-7002
> URL: https://issues.apache.org/jira/browse/HDDS-7002
> Project: Apache Ozone
> Issue Type: Bug
> Components: Native
> Affects Versions: 1.2.1
> Environment: ozone 1.2.1
> hdfs 3.1.3
> Reporter: Dmitry Kravchuk
> Priority: Critical
> Fix For: 1.2.1
>
>
> I'm following with documentation guide here [https://hadoop.apache.org/ozone/docs/0.4.1-alpha/start/onprem.html] and always fails after trying to start datanodes - they are always in safemode. Can somebody help me out with understanding how to scale ozone up to more than 1 datanode?
>
> Here is my $OZONE_HOME/etc/hadoop/ozone-env.sh:
> {code:sh}
> #
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.# Set Ozone-specific environment variables here.# Enable core dump when crash in C++
> ulimit -c unlimited# Many of the options here are built from the perspective that users
> # may want to provide OVERWRITING values on the command line.
> # For example:
> #
> # JAVA_HOME=/usr/java/testing ozone fs -ls
> #
> # Therefore, the vast majority (BUT NOT ALL!) of these defaults
> # are configured for substitution and not append. If append
> # is preferable, modify this file accordingly.###
> # Generic settings
> #### Technically, the only required environment variable is JAVA_HOME.
> # All others are optional. However, the defaults are probably not
> # preferred. Many sites configure these options outside of Ozone,
> # such as in /etc/profile.d# The java implementation to use. By default, this environment
> # variable is REQUIRED on ALL platforms except OS X!
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64/jre# /opt/ozone/ozone-1.2.1
> # Location of Ozone. By default, Ozone will attempt to determine
> # this location based upon its execution path.
> export OZONE_HOME=/opt/ozone/ozone-1.2.1# Location of Ozone's configuration information. i.e., where this
> # file is living. If this is not defined, Ozone will attempt to
> # locate it based upon its execution path.
> #
> # NOTE: It is recommend that this variable not be set here but in
> # /etc/profile.d or equivalent. Some options (such as
> # --config) may react strangely otherwise.
> #
> export OZONE_CONF_DIR=${OZONE_HOME}/etc/hadoop# The maximum amount of heap to use (Java -Xmx). If no unit
> # is provided, it will be converted to MB. Daemons will
> # prefer any Xmx setting in their respective _OPT variable.
> # There is no default; the JVM will autoscale based upon machine
> # memory size.
> # export OZONE_HEAPSIZE_MAX=# The minimum amount of heap to use (Java -Xms). If no unit
> # is provided, it will be converted to MB. Daemons will
> # prefer any Xms setting in their respective _OPT variable.
> # There is no default; the JVM will autoscale based upon machine
> # memory size.
> # export OZONE_HEAPSIZE_MIN=# Extra Java runtime options for all Ozone commands. We don't support
> # IPv6 yet/still, so by default the preference is set to IPv4.
> # export OZONE_OPTS="-Djava.net.preferIPv4Stack=true"
> # For Kerberos debugging, an extended option set logs more information
> export OZONE_OPTS="-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled"
> #"-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"# Some parts of the shell code may do special things dependent upon
> # the operating system. We have to set this here. See the next
> # section as to why....
> export OZONE_OS_TYPE=${OZONE_OS_TYPE:-$(uname -s)}# Extra Java runtime options for some Ozone commands
> # and clients (e.g., ozone sh). These get appended to OZONE_OPTS for
> # such commands. In most cases, this should be left empty and
> # let users supply it on the command line.
> # export OZONE_CLIENT_OPTS=""#
> # A note about classpaths.
> #
> # By default, Apache Ozone overrides Java's CLASSPATH
> # environment variable. It is configured such
> # that it starts out blank with new entries added after passing
> # a series of checks (file/dir exists, not already listed aka
> # de-deduplication). During de-deduplication, wildcards and/or
> # directories are *NOT* expanded to keep it simple. Therefore,
> # if the computed classpath has two specific mentions of
> # awesome-methods-1.0.jar, only the first one added will be seen.
> # If two directories are in the classpath that both contain
> # awesome-methods-1.0.jar, then Java will pick up both versions.# An additional, custom CLASSPATH. Site-wide configs should be
> # handled via the shellprofile functionality, utilizing the
> # ozone_add_classpath function for greater control and much
> # harder for apps/end-users to accidentally override.
> # Similarly, end users should utilize ${HOME}/.ozonerc .
> # This variable should ideally only be used as a short-cut,
> # interactive way for temporary additions on the command line.
> # export OZONE_CLASSPATH="/some/cool/path/on/your/machine"# Should OZONE_CLASSPATH be first in the official CLASSPATH?
> # export OZONE_USER_CLASSPATH_FIRST="yes"# If OZONE_USE_CLIENT_CLASSLOADER is set, OZONE_CLASSPATH and
> # OZONE_USER_CLASSPATH_FIRST are ignored.
> # export OZONE_USE_CLIENT_CLASSLOADER=true###
> # Options for remote shell connectivity
> #### There are some optional components of ozone that allow for
> # command and control of remote hosts. For example,
> # start-ozone.sh will attempt to bring up all OMs, SCMs, DNs, etc.# Options to pass to SSH when one of the "log into a host and
> # start/stop daemons" scripts is executed
> # export OZONE_SSH_OPTS="-o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10s"# The built-in ssh handler will limit itself to 10 simultaneous connections.
> # For pdsh users, this sets the fanout size ( -f )
> # Change this to increase/decrease as necessary.
> # export OZONE_SSH_PARALLEL=10# Filename which contains all of the hosts for any remote execution
> # helper scripts # such as workers.sh, start-ozone.sh, etc.
> # export OZONE_WORKERS="${OZONE_CONF_DIR}/workers"###
> # Options for all daemons
> ###
> ##
> # Many options may also be specified as Java properties. It is
> # very common, and in many cases, desirable, to hard-set these
> # in daemon _OPTS variables. Where applicable, the appropriate
> # Java property is also identified. Note that many are re-used
> # or set differently in certain contexts (e.g., secure vs
> # non-secure)
> ## Where (primarily) daemon log files are stored.
> # ${OZONE_HOME}/logs by default.
> # Java property: hadoop.log.dir
> # export OZONE_LOG_DIR=${OZONE_HOME}/logs# A string representing this instance of Ozone. $USER by default.
> # This is used in writing log and pid files, so keep that in mind!
> # Java property: hadoop.id.str
> # export OZONE_IDENT_STRING=$USER# How many seconds to pause after stopping a daemon
> # export OZONE_STOP_TIMEOUT=5# Where pid files are stored. /tmp by default.
> export OZONE_PID_DIR=/data/disk1/meta# Default log4j setting for interactive commands
> # Java property: hadoop.root.logger
> # export OZONE_ROOT_LOGGER=INFO,console# Default log4j setting for daemons spawned explicitly by
> # --daemon option of ozone command.
> # Java property: hadoop.root.logger
> # export OZONE_DAEMON_ROOT_LOGGER=INFO,RFA# Default log level and output location for security-related messages.
> # You will almost certainly want to change this on a per-daemon basis via
> # the Java property (i.e., -Dhadoop.security.logger=foo).
> # Java property: hadoop.security.logger
> # export OZONE_SECURITY_LOGGER=INFO,NullAppender# Default process priority level
> # Note that sub-processes will also run at this level!
> # export OZONE_NICENESS=0# Default name for the service level authorization file
> # Java property: hadoop.policy.file
> # export OZONE_POLICYFILE="hadoop-policy.xml"#
> # NOTE: this is not used by default! <-----
> # You can define variables right here and then re-use them later on.
> # For example, it is common to use the same garbage collection settings
> # for all the daemons. So one could define:
> #
> # export OZONE_GC_SETTINGS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
> #
> # .. and then use it when setting OZONE_OM_OPTS, etc. below###
> # Secure/privileged execution
> ####
> # Out of the box, Ozone uses jsvc from Apache Commons to launch daemons
> # on privileged ports. This functionality can be replaced by providing
> # custom functions. See ozone-functions.sh for more information.
> ## The jsvc implementation to use. Jsvc is required to run secure datanodes
> # that bind to privileged ports to provide authentication of data transfer
> # protocol. Jsvc is not required if SASL is configured for authentication of
> # data transfer protocol using non-privileged ports.
> # export JSVC_HOME=/usr/bin#
> # This directory contains pids for secure and privileged processes.
> #export OZONE_SECURE_PID_DIR=${OZONE_PID_DIR}#
> # This directory contains the logs for secure and privileged processes.
> # Java property: hadoop.log.dir
> # export OZONE_SECURE_LOG=${OZONE_LOG_DIR}#
> # When running a secure daemon, the default value of OZONE_IDENT_STRING
> # ends up being a bit bogus. Therefore, by default, the code will
> # replace OZONE_IDENT_STRING with OZONE_xx_SECURE_USER. If one wants
> # to keep OZONE_IDENT_STRING untouched, then uncomment this line.
> # export OZONE_SECURE_IDENT_PRESERVE="true"###
> # Ozone Manager specific parameters
> ###
> # Specify the JVM options to be used when starting the Ozone Manager.
> # These options will be appended to the options specified as OZONE_OPTS
> # and therefore may override any similar flags set in OZONE_OPTS
> #
> # export OZONE_OM_OPTS=""###
> # Ozone DataNode specific parameters
> ###
> # Specify the JVM options to be used when starting Ozone DataNodes.
> # These options will be appended to the options specified as OZONE_OPTS
> # and therefore may override any similar flags set in OZONE_OPTS
> #
> # export OZONE_DATANODE_OPTS=""###
> # HDFS StorageContainerManager specific parameters
> ###
> # Specify the JVM options to be used when starting the HDFS Storage Container Manager.
> # These options will be appended to the options specified as OZONE_OPTS
> # and therefore may override any similar flags set in OZONE_OPTS
> #
> # export OZONE_SCM_OPTS=""###
> # Advanced Users Only!
> ####
> # When building Ozone, one can add the class paths to the commands
> # via this special env var:
> # export OZONE_ENABLE_BUILD_PATHS="true"#
> # To prevent accidents, shell commands be (superficially) locked
> # to only allow certain users to execute certain subcommands.
> # It uses the format of (command)_(subcommand)_USER.
> #
> # For example, to limit who can execute the om command,
> export OZONE_OM_USER=rootHDFS_DATANODE_USER=root
> HDFS_SCM_USER=root
> HDFS_OM_USER=root{code}
> Here is my $OZONE_HOME/etc/hadoop/ozone-site.xml
> {code:xml}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <configuration>
> <property>
> <name>ozone.om.address</name>
> <value>hadoop-master-3.my.company.com</value>
> <tag>OM, REQUIRED</tag>
> <description>
> The address of the Ozone OM service. This allows clients to discover
> the address of the OM.
> </description>
> </property>
> <property>
> <name>ozone.metadata.dirs</name>
> <value>/data/disk1/meta</value>
> <tag>OZONE, OM, SCM, CONTAINER, STORAGE, REQUIRED</tag>
> <description>
> This setting is the fallback location for SCM, OM, Recon and DataNodes
> to store their metadata. This setting may be used only in test/PoC
> clusters to simplify configuration. For production clusters or any time you care about performance, it is
> recommended that ozone.om.db.dirs, ozone.scm.db.dirs and
> dfs.container.ratis.datanode.storage.dir be configured separately.
> </description>
> </property>
> <property>
> <name>ozone.scm.client.address</name>
> <value>hadoop-master-3.my.company.com</value>
> <tag>OZONE, SCM, REQUIRED</tag>
> <description>
> The address of the Ozone SCM client service. This is a required setting. It is a string in the host:port format. The port number is optional
> and defaults to 9860.
> </description>
> </property>
> <property>
> <name>ozone.scm.names</name>
> <value>hadoop-master-3.my.company.com</value>
> <tag>OZONE, REQUIRED</tag>
> <description>
> The value of this property is a set of DNS | DNS:PORT | IP
> Address | IP:PORT. Written as a comma separated string. e.g. scm1,
> scm2:8020, 7.7.7.7:7777.
> This property allows datanodes to discover where SCM is, so that
> datanodes can send heartbeat to SCM.
> </description>
> </property>
> <property>
> <name>ozone.scm.datanode.id.dir</name>
> <value>/data/disk1/meta/node</value>
> </property>
> <property>
> <name>ozone.scm.datanode.address</name>
> <value>hadoop-master-3.my.company.com</value>
> </property>
> </configuration> {code}
>
> Here is my $OZONE_HOME/etc/hadoop/workers:
> {code:sh}
> hadoop-dn-01.my.company.com
> {code}
>
> Step to reproduce:
> * cd $OZONE_HOME
> * bin/ozone scm --init
> * bin/ozone --daemon start scm
> * bin/ozone om --init
> * bin/ozone --daemon start om
> * sbin/stop-ozone.sh
> * sbin/start-ozone.sh
> * bin/ozone --daemon start recon
> Then I have 0 healthy nodes in recon and 1 healthy node in safe mode with command bin/ozone admin datanode list. I faced this strage problem cause on prem deployment documentation does not have any instructions how should condigs at datanodes hosts look like. Where I am wrong and how to correnctly deploy ozone in cluster mode with scm, om and recon at master host and datanodes at another multiple hosts?
>
> Thank you.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org