[ https://issues.apache.org/jira/browse/AMBARI-11630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hurley updated AMBARI-11630:
-------------------------------------
Attachment: AMBARI-11630.patch
> Falcon Server Seems To Have Hadoop 2.2 Paths After Upgrade
> ----------------------------------------------------------
>
> Key: AMBARI-11630
> URL: https://issues.apache.org/jira/browse/AMBARI-11630
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.1.0
> Reporter: Jonathan Hurley
> Assignee: Jonathan Hurley
> Priority: Critical
> Fix For: 2.1.0
>
> Attachments: AMBARI-11630.patch
>
>
> After an upgrade from HDp 2.2 to HDP 2.3, Falcon seems to have a mixture of HDP 2.2 and 2.3 paths in its process path:
> {noformat}
> [root@jhurley-hdp22-ru-6 ~]# ps aux | grep falcon
> falcon 16231 0.2 6.0 3623356 455580 ? Sl 14:34 0:18 /usr/jdk64/jdk1.8.0_40/bin/java -Xmx1024m -noverify -Dfalcon.embeddedmq=True -Dfalcon.emeddedmq.port=61616 -Dfalcon.log.dir=/var/log/falcon -Dfalcon.embeddedmq.data=/hadoop/falcon/embeddedmq/data -Dfalcon.home=/usr/hdp/current/falcon-server -Dconfig.location=/usr/hdp/current/falcon-server/conf -Dfalcon.app.type=falcon -Dfalcon.catalog.service.enabled= -cp /usr/hdp/current/falcon-server/conf:/usr/hdp/2.2.7.0-2808/hadoop/conf:/usr/hdp/2.2.7.0-2808/hadoop/lib/*:/usr/hdp/2.2.7.0-2808/hadoop/.//*:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/./:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-hdfs/.//*:/usr/hdp/2.2.7.0-2808/hadoop-yarn/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-yarn/.//*:/usr/hdp/2.2.7.0-2808/hadoop-mapreduce/lib/*:/usr/hdp/2.2.7.0-2808/hadoop-mapreduce/.//*::/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.2.7.0-2808/tez/*:/usr/hdp/2.2.7.0-2808/tez/lib/*:/etc/tez/conf:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classes:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hdp/current/falcon-server/libext/* org.apache.falcon.Main -app /usr/hdp/current/falcon-server/server/webapp/falcon -port 15000
> root 28434 0.0 0.0 103252 860 pts/5 S+ 16:44 0:00 grep falcon
> [root@jhurley-hdp22-ru-6 ~]# ll /usr/hdp/current | grep falcon
> lrwxrwxrwx. 1 root root 28 May 29 03:30 falcon-client -> /usr/hdp/2.2.7.0-2808/falcon
> lrwxrwxrwx. 1 root root 28 May 29 14:34 falcon-server -> /usr/hdp/2.3.0.0-2162/falcon
> {noformat}
> A restart of Falcon post-upgrade corrects these problems, so it seems like it's an issue only during upgrade.
> Also, {{falcon-client}} is not upgraded as part of the upgrade process. This may be covered by the {{hdp-select all}} work capture in another Jira, but falcon-server may still need the client updated ahead of that.
> The root cause of this is the fact that Ambari is not setting {{HADOOP_HOME}} when invoking Falcon. Under most circumstances, this is OK since Falcon's {{falcon-config.sh}} script will then try to figure it out with {{which hadoop}}:
> {code:title=falcon-config.sh HDP 2.2}
> HADOOPDIR=`which hadoop`
> if [ "$HADOOPDIR" != "" ]; then
> echo "Hadoop is installed, adding hadoop classpath to falcon classpath"
> FALCONCPPATH="${FALCONCPPATH}:`hadoop classpath`"
> elif [ "$HADOOP_HOME" != "" ]; then
> echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/hadoop classpath' into falcon classpath"
> FALCONCPPATH="${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`"
> else
> echo "Could not find installed hadoop and HADOOP_HOME is not set."
> echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
> FALCONCPPATH="${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
> fi
> {code}
> The problem in HDP 2.2 is that it will always find the {{HADOOPDIR}} first and never pickup {{HADOOP_HOME}}. In an upgraded environment, it will always use the 2.2 libs.
> This appears to heave been fixed in HDP 2.3:
> {code:title=falcon-config.sh HDP 2.3}
> FALCONCPPATH="$FALCON_CONF"
> HADOOPDIR=`which hadoop`
> if [ "$HADOOP_HOME" != "" ]; then
> echo "Hadoop home is set, adding libraries from '${HADOOP_HOME}/bin/hadoop classpath' into falcon classpath"
> FALCONCPPATH="${FALCONCPPATH}:`${HADOOP_HOME}/bin/hadoop classpath`"
> elif [ "$HADOOPDIR" != "" ]; then
> echo "Hadoop is installed, adding hadoop classpath to falcon classpath"
> FALCONCPPATH="${FALCONCPPATH}:`hadoop classpath`"
> else
> echo "Could not find installed hadoop and HADOOP_HOME is not set."
> echo "Using the default jars bundled in ${BASEDIR}/hadooplibs/"
> FALCONCPPATH="${FALCONCPPATH}:${BASEDIR}/hadooplibs/*"
> fi
> {code}
> So we just need to ensure we pass in {{HADOOP_HOME}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)