You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "hznanhai (JIRA)" <ji...@apache.org> on 2015/10/13 10:42:06 UTC
[jira] [Updated] (AMBARI-10992) Ambari is too slow to start/restart
components
[ https://issues.apache.org/jira/browse/AMBARI-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
hznanhai updated AMBARI-10992:
------------------------------
Epic Name: Ambari is too slow to start/restart (was: Ambari is too slow to start/restart components)
> Ambari is too slow to start/restart components
> ----------------------------------------------
>
> Key: AMBARI-10992
> URL: https://issues.apache.org/jira/browse/AMBARI-10992
> Project: Ambari
> Issue Type: Epic
> Affects Versions: 2.0.0
> Reporter: Andrew Onischuk
> Assignee: Andrew Onischuk
> Priority: Critical
> Fix For: 2.1.0
>
>
> *PROBLEM REASONING*
> This log of HiveServer2 restart (which took 5 minutes) brightly describes what takes the most time usually for no big reason:
> {noformat}
> 014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
> 2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
> 2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
> 2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
> 2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
> 2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
> 2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...}
> 2014-12-10 17:14:50,979 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml
> 2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
> 2014-12-10 17:14:50,983 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
> 2014-12-10 17:14:51,114 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
> 2014-12-10 17:14:51,169 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
> 2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
> 2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist
> 2014-12-10 17:14:51,312 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
> 2014-12-10 17:14:51,367 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
> 2014-12-10 17:14:51,423 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
> 2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
> 2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
> 2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
> {noformat}
> The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node load CPU/RAM etc. Mostly this is because every time call is done new jvm is created and probably due to some DN/NN communication is redone. Since we have more and more this calls triggered with every release, this became a problem now.
> *BASIC SOLUTION DESCRIPTION*
> Instead of copying files to hdfs and creating directories one by one, by calling countless number of ‘hadoop fs’ calls, we can do one single execution of the jar file which will do all the creating it is asked.
> For example:
> {noformat}
> hadoop -fs -mkdir /a
> hadoop -fs -chmod 755 /a
> hadoop -fs -chown ambari-qa /b
> hadoop -fs -copyFromLocal /x /y
> {noformat}
> Can be replaced with
> resources.json like this which is passed to our new jar (below):
> {noformat}
> {
> {
> “path”:”/a”
> “type”: directory”
> “mode”: “755”
> },
> {
> “path”:”/b”
> “type”: directory”
> “owner”: “ambari-qa”
> },
> {
> “path”:”/y”
> “type”: directory”
> “source”: “/x”
> },
> }
> {noformat}
> which will handle all that only in one single time-expensive call:
> *hadoop jar create-hdfs-resources.jar resources.json*
> For synchronization between nodes (to know who and when executes this time expensive calls) ZK is used for more details please read the doc below
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)