You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Andrew Onischuk (JIRA)" <ji...@apache.org> on 2017/06/13 13:58:00 UTC
[jira] [Updated] (AMBARI-21239) Ambari leaves a lot of processes
from Hive alert on the machine
[ https://issues.apache.org/jira/browse/AMBARI-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Onischuk updated AMBARI-21239:
-------------------------------------
Attachment: AMBARI-21239.patch
> Ambari leaves a lot of processes from Hive alert on the machine
> ---------------------------------------------------------------
>
> Key: AMBARI-21239
> URL: https://issues.apache.org/jira/browse/AMBARI-21239
> Project: Ambari
> Issue Type: Bug
> Reporter: Andrew Onischuk
> Assignee: Andrew Onischuk
> Fix For: 2.5.2
>
> Attachments: AMBARI-21239.patch
>
>
> I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
> the database VM storing the metadata after the cluster had been runing for
> almost 2 weeks without issues. What I observed is that the instance in a few
> hours got really slow. Upon investigation found hundreds of Beeline processes
> like this:
>
>
> ambari-+ 3439770 3439769 0 06:19 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3439814 3439770 0 06:19 ? 00:00:42 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e
> ambari-+ 3439816 3439770 0 06:19 ? 00:00:00 awk {print}
> ambari-+ 3439817 3439770 0 06:19 ? 00:00:00 grep -i -e Connection refused -e Invalid URL
> ambari-+ 3440645 3440644 0 06:22 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3440685 3440645 0 06:22 ? 00:00:42 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e
> ambari-+ 3440686 3440645 0 06:22 ? 00:00:00 awk {print}
> ambari-+ 3440687 3440645 0 06:22 ? 00:00:00 grep -i -e Connection refused -e Invalid URL
> ambari-+ 3441185 3441184 0 06:25 ? 00:00:00 -bash -c export PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary' -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3441224 3441185 0 06:25 ? 00:00:46 /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx1024m -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://ctr-e133-1493418528701-31316-01-000008.hwx.site:10500/;transportMode=binary -e
>
> Upon taking a count of the processes with org.apache.hive.beeline.BeeLine in
> them and owned by ambari found 309 processes.
> The cluster is accessible at 172.27.15.4, unfortunately the Ambari UI isn't
> accessible as the DB host serving the metadata went down.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)