You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Andrew Onischuk (JIRA)" <ji...@apache.org> on 2015/05/07 10:33:59 UTC
[jira] [Updated] (AMBARI-10992) Ambari is too slow to start/restart
components
[ https://issues.apache.org/jira/browse/AMBARI-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Onischuk updated AMBARI-10992:
-------------------------------------
Description:
*PROBLEM REASONING*
This log of HiveServer2 restart (which took 5 minutes) brightly describes what takes the most time usually for no big reason:
{noformat}
014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...}
2014-12-10 17:14:50,979 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml
2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2014-12-10 17:14:50,983 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
2014-12-10 17:14:51,114 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
2014-12-10 17:14:51,169 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist
2014-12-10 17:14:51,312 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
2014-12-10 17:14:51,367 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
2014-12-10 17:14:51,423 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
{noformat}
The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node load CPU/RAM etc. Mostly this is because every time call is done new jvm is created and probably due to some DN/NN communication is redone. Since we have more and more this calls triggered with every release, this became a problem now.
*BASIC SOLUTION DESCRIPTION*
Instead of copying files to hdfs and creating directories one by one, by calling countless number of ‘hadoop fs’ calls, we can do one single execution of the jar file which will do all the creating it is asked.
For example:
{noformat}
hadoop -fs -mkdir /a
hadoop -fs -chmod 755 /a
hadoop -fs -chown ambari-qa /b
hadoop -fs -copyFromLocal /x /y
{noformat}
Can be replaced with
resources.json like this which is passed to our new jar (below):
{noformat}
{
{
“path”:”/a”
“type”: directory”
“mode”: “755”
},
{
“path”:”/b”
“type”: directory”
“owner”: “ambari-qa”
},
{
“path”:”/y”
“type”: directory”
“source”: “/x”
},
}
{noformat}
which will handle all that only in one single time-expensive call:
*hadoop jar create-hdfs-resources.jar resources.json*
For synchronization between nodes ZK for more details please read the doc below
was:
*PROBLEM REASONING*
This log of HiveServer2 restart (which took 5 minutes) brightly describes what takes the most time usually for no big reason:
{noformat}
014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...}
2014-12-10 17:14:50,979 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml
2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2014-12-10 17:14:50,983 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
2014-12-10 17:14:51,114 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
2014-12-10 17:14:51,169 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist
2014-12-10 17:14:51,312 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
2014-12-10 17:14:51,367 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
2014-12-10 17:14:51,423 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
{noformat}
The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node load CPU/RAM etc. Mostly this is because every time call is done new jvm is created and probably due to some DN/NN communication is redone. Since we have more and more this calls triggered with every release, this became a problem now.
*BASIC SOLUTION DESCRIPTION*
Instead of copying files to hdfs and creating directories one by one, by calling countless number of ‘hadoop fs’ calls, we can do one single execution of the jar file which will do all the creating it is asked.
For example:
{noformat}
hadoop -fs -mkdir /a
hadoop -fs -chmod 755 /a
hadoop -fs -chown ambari-qa /b
hadoop -fs -copyFromLocal /x /y
{noformat}
Can be replaced with
resources.json like this which is passed to our new jar (below):
{noformat}
{
{
“path”:”/a”
“type”: directory”
“mode”: “755”
},
{
“path”:”/b”
“type”: directory”
“owner”: “ambari-qa”
},
{
“path”:”/y”
“type”: directory”
“source”: “/x”
},
}
{noformat}
which will handle all that in one single call:
*hadoop jar create-hdfs-resources.jar resources.json*
For synchronization between nodes ZK for more details please read the doc below
> Ambari is too slow to start/restart components
> ----------------------------------------------
>
> Key: AMBARI-10992
> URL: https://issues.apache.org/jira/browse/AMBARI-10992
> Project: Ambari
> Issue Type: Epic
> Reporter: Andrew Onischuk
> Assignee: Andrew Onischuk
> Priority: Critical
> Fix For: Ambari-2.1
>
>
> *PROBLEM REASONING*
> This log of HiveServer2 restart (which took 5 minutes) brightly describes what takes the most time usually for no big reason:
> {noformat}
> 014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
> 2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:29,516 - Execute['hadoop --con014-12-10 17:09:29,060 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:09:29,073 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:09:46,301 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz': No such file or directory
> 2014-12-10 17:09:46,301 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:09:46,303 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/hive && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:10:29,989 - CopyFromLocal['/usr/hdp/current/hive-client/hive.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/hive', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:10:30,017 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:10:48,275 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hive-client/hive.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/hive'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:07,134 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:07,135 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:16,533 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:16,534 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/hive/hive.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:29,515 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:11:29,516 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
> 2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
> 2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
> 2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
> 2014-12-10 17:14:50,862 - XmlConfig['webhcat-site.xml'] {'owner': 'hcat', 'group': 'hadoop', 'conf_dir': '/etc/hive-webhcat/conf', 'configuration_attributes': ..., 'configurations': ...}
> 2014-12-10 17:14:50,979 - Generating config: /etc/hive-webhcat/conf/webhcat-site.xml
> 2014-12-10 17:14:50,980 - File['/etc/hive-webhcat/conf/webhcat-site.xml'] {'owner': 'hcat', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
> 2014-12-10 17:14:50,983 - Writing File['/etc/hive-webhcat/conf/webhcat-site.xml'] because it doesn't exist
> 2014-12-10 17:14:51,114 - Changing owner for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hcat
> 2014-12-10 17:14:51,169 - Changing group for /etc/hive-webhcat/conf/webhcat-site.xml from 0 to hadoop
> 2014-12-10 17:14:51,221 - File['/etc/hive-webhcat/conf/webhcat-env.sh'] {'content': InlineTemplate(...), 'owner': 'hcat', 'group': 'hadoop'}
> 2014-12-10 17:14:51,222 - Writing File['/etc/hive-webhcat/conf/webhcat-env.sh'] because it doesn't exist
> 2014-12-10 17:14:51,312 - Changing owner for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hcat
> 2014-12-10 17:14:51,367 - Changing group for /etc/hive-webhcat/conf/webhcat-env.sh from 0 to hadoop
> 2014-12-10 17:14:51,423 - Execute['env HADOOP_HOME=/usr/hdp/current/hadoop-client /usr/hdp/current/hive-webhcat/sbin/webhcat_server.sh start'] {'not_if': 'ls /var/run/webhcat/webhcat.pid >/dev/null 2>&1 && ps -p `cat /var/run/webhcat/webhcat.pid` >/dev/null 2>&1', 'user': 'hcat'}fig /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:11:45,791 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz': No such file or directory
> 2014-12-10 17:11:45,791 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:11:45,794 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/pig && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:12:31,703 - CopyFromLocal['/usr/hdp/current/pig-client/pig.tar.gz'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/pig', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:12:31,703 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:12:49,508 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/pig-client/pig.tar.gz hdfs:///hdp/apps/2.2.0.0-2041/pig'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:09,506 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:09,507 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:18,968 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:18,969 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/pig/pig.tar.gz'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:32,936 - ExecuteHadoop['fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hcat', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:13:32,937 - Execute['hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': True, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hcat', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:13:52,891 - ls: `hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar': No such file or directory
> 2014-12-10 17:13:52,892 - HdfsDirectory['hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '', 'mode': 0555, 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create']}
> 2014-12-10 17:13:52,904 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; hadoop --config /etc/hadoop/conf fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:03,832 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chmod 555 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce && hadoop --config /etc/hadoop/conf fs -chown hdfs hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] due to not_if
> 2014-12-10 17:14:03,833 - CopyFromLocal['/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar'] {'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'group': 'hadoop', 'hdfs_user': 'hdfs', 'owner': 'hdfs', 'kinnit_if_needed': '', 'dest_dir': 'hdfs:///hdp/apps/2.2.0.0-2041/mapreduce', 'hadoop_conf_dir': '/etc/hadoop/conf', 'mode': 0444}
> 2014-12-10 17:14:03,836 - ExecuteHadoop['fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'not_if': "/usr/bin/sudo su hdfs -l -s /bin/bash -c 'export {ENV_PLACEHOLDER} > /dev/null ; PATH=$PATH:/usr/hdp/current/hadoop-client/bin hadoop fs -ls hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'", 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:12,682 - Execute['hadoop --config /etc/hadoop/conf fs -copyFromLocal /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar hdfs:///hdp/apps/2.2.0.0-2041/mapreduce'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:22,350 - ExecuteHadoop['fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:22,352 - Execute['hadoop --config /etc/hadoop/conf fs -chown hdfs:hadoop hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:34,163 - ExecuteHadoop['fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'user': 'hdfs', 'conf_dir': '/etc/hadoop/conf'}
> 2014-12-10 17:14:34,164 - Execute['hadoop --config /etc/hadoop/conf fs -chmod 444 hdfs:///hdp/apps/2.2.0.0-2041/mapreduce/hadoop-streaming.jar'] {'logoutput': False, 'try_sleep': 0, 'environment': ..., 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
> 2014-12-10 17:14:50,851 - Could not find file: /usr/hdp/current/sqoop-client/sqoop.tar.gz
> {noformat}
> The problem is ‘hadoop fs’ calls. One call takes 3-30 seconds depending on node load CPU/RAM etc. Mostly this is because every time call is done new jvm is created and probably due to some DN/NN communication is redone. Since we have more and more this calls triggered with every release, this became a problem now.
> *BASIC SOLUTION DESCRIPTION*
> Instead of copying files to hdfs and creating directories one by one, by calling countless number of ‘hadoop fs’ calls, we can do one single execution of the jar file which will do all the creating it is asked.
> For example:
> {noformat}
> hadoop -fs -mkdir /a
> hadoop -fs -chmod 755 /a
> hadoop -fs -chown ambari-qa /b
> hadoop -fs -copyFromLocal /x /y
> {noformat}
> Can be replaced with
> resources.json like this which is passed to our new jar (below):
> {noformat}
> {
> {
> “path”:”/a”
> “type”: directory”
> “mode”: “755”
> },
> {
> “path”:”/b”
> “type”: directory”
> “owner”: “ambari-qa”
> },
> {
> “path”:”/y”
> “type”: directory”
> “source”: “/x”
> },
> }
> {noformat}
> which will handle all that only in one single time-expensive call:
> *hadoop jar create-hdfs-resources.jar resources.json*
> For synchronization between nodes ZK for more details please read the doc below
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)