You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2014/10/26 17:28:29 UTC
Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/
-----------------------------------------------------------
Review request for Ambari and Dmytro Sen.
Bugs: AMBARI-7978
https://issues.apache.org/jira/browse/AMBARI-7978
Repository: ambari
Description
-------
When installing HDP, change the log directory of Falcon to non-default
for ex:
>From /var/log/falcon TO /grid/0/log/falcon
Install of falcon client fails and HDP cluster install fails with error that
/grid/0/log directory could not be found.
Expected:
log directory should be created if it does not exist
Log:
stderr:
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
stdout:
2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
2014-10-26 02:45:42,107 - Modifying group hadoop
2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
2014-10-26 02:45:42,229 - Modifying group nobody
2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
2014-10-26 02:45:42,318 - Modifying group users
2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
2014-10-26 02:45:42,423 - Modifying group nagios
2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
2014-10-26 02:45:42,518 - Modifying group knox
2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,616 - Modifying user hive
2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:42,730 - Modifying user oozie
2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
2014-10-26 02:45:42,830 - Modifying user nobody
2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,922 - Modifying user nagios
2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,014 - Modifying user ambari-qa
2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,116 - Modifying user flume
2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,205 - Modifying user hdfs
2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,290 - Modifying user knox
2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,379 - Modifying user storm
2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,470 - Modifying user mapred
2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,564 - Modifying user hbase
2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,659 - Modifying user tez
2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,753 - Modifying user zookeeper
2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,845 - Modifying user falcon
2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,935 - Modifying user sqoop
2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,022 - Modifying user yarn
2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,120 - Modifying user hcat
2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,483 - Package['unzip'] {}
2014-10-26 02:45:44,639 - Skipping installing existent package unzip
2014-10-26 02:45:44,640 - Package['curl'] {}
2014-10-26 02:45:44,734 - Skipping installing existent package curl
2014-10-26 02:45:44,735 - Package['hdp-select'] {}
2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
Diff: https://reviews.apache.org/r/27210/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/#review58562
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Sen
On Окт. 26, 2014, 4:33 п.п., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27210/
> -----------------------------------------------------------
>
> (Updated Окт. 26, 2014, 4:33 п.п.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Mahadev Konar.
>
>
> Bugs: AMBARI-7978
> https://issues.apache.org/jira/browse/AMBARI-7978
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing HDP, change the log directory of Falcon to non-default
> for ex:
> From /var/log/falcon TO /grid/0/log/falcon
>
> Install of falcon client fails and HDP cluster install fails with error that
> /grid/0/log directory could not be found.
>
> Expected:
> log directory should be created if it does not exist
>
>
>
> Log:
> stderr:
> 2014-10-26 02:45:59,612 - Error while executing command 'install':
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
> self.configure(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
> falcon('client', action='config')
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
> owner=params.falcon_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
> raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
> Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
> stdout:
> 2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
> 2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
> 2014-10-26 02:45:42,107 - Modifying group hadoop
> 2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
> 2014-10-26 02:45:42,229 - Modifying group nobody
> 2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
> 2014-10-26 02:45:42,318 - Modifying group users
> 2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
> 2014-10-26 02:45:42,423 - Modifying group nagios
> 2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
> 2014-10-26 02:45:42,518 - Modifying group knox
> 2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:42,616 - Modifying user hive
> 2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:42,730 - Modifying user oozie
> 2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
> 2014-10-26 02:45:42,830 - Modifying user nobody
> 2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:42,922 - Modifying user nagios
> 2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:43,014 - Modifying user ambari-qa
> 2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,116 - Modifying user flume
> 2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,205 - Modifying user hdfs
> 2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,290 - Modifying user knox
> 2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,379 - Modifying user storm
> 2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,470 - Modifying user mapred
> 2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,564 - Modifying user hbase
> 2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:43,659 - Modifying user tez
> 2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,753 - Modifying user zookeeper
> 2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,845 - Modifying user falcon
> 2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,935 - Modifying user sqoop
> 2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:44,022 - Modifying user yarn
> 2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:44,120 - Modifying user hcat
> 2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
> 2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
> 2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
> 2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
> 2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
> 2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
> 2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
> 2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
> 2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
> 2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
> 2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
> 2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
> 2014-10-26 02:45:44,483 - Package['unzip'] {}
> 2014-10-26 02:45:44,639 - Skipping installing existent package unzip
> 2014-10-26 02:45:44,640 - Package['curl'] {}
> 2014-10-26 02:45:44,734 - Skipping installing existent package curl
> 2014-10-26 02:45:44,735 - Package['hdp-select'] {}
> 2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
> 2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
> 2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
> 2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
> 2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
> 2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
> 2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
> 2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
> 2014-10-26 02:45:59,612 - Error while executing command 'install':
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
> self.configure(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
> falcon('client', action='config')
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
> owner=params.falcon_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
> raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
> Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
>
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
>
> Diff: https://reviews.apache.org/r/27210/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>
Re: Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/
-----------------------------------------------------------
(Updated Oct. 26, 2014, 4:34 p.m.)
Review request for Ambari, Dmitro Lisnichenko and Dmytro Sen.
Bugs: AMBARI-7978
https://issues.apache.org/jira/browse/AMBARI-7978
Repository: ambari
Description
-------
When installing HDP, change the log directory of Falcon to non-default
for ex:
>From /var/log/falcon TO /grid/0/log/falcon
Install of falcon client fails and HDP cluster install fails with error that
/grid/0/log directory could not be found.
Expected:
log directory should be created if it does not exist
Log:
stderr:
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
stdout:
2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
2014-10-26 02:45:42,107 - Modifying group hadoop
2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
2014-10-26 02:45:42,229 - Modifying group nobody
2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
2014-10-26 02:45:42,318 - Modifying group users
2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
2014-10-26 02:45:42,423 - Modifying group nagios
2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
2014-10-26 02:45:42,518 - Modifying group knox
2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,616 - Modifying user hive
2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:42,730 - Modifying user oozie
2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
2014-10-26 02:45:42,830 - Modifying user nobody
2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,922 - Modifying user nagios
2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,014 - Modifying user ambari-qa
2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,116 - Modifying user flume
2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,205 - Modifying user hdfs
2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,290 - Modifying user knox
2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,379 - Modifying user storm
2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,470 - Modifying user mapred
2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,564 - Modifying user hbase
2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,659 - Modifying user tez
2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,753 - Modifying user zookeeper
2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,845 - Modifying user falcon
2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,935 - Modifying user sqoop
2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,022 - Modifying user yarn
2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,120 - Modifying user hcat
2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,483 - Package['unzip'] {}
2014-10-26 02:45:44,639 - Skipping installing existent package unzip
2014-10-26 02:45:44,640 - Package['curl'] {}
2014-10-26 02:45:44,734 - Skipping installing existent package curl
2014-10-26 02:45:44,735 - Package['hdp-select'] {}
2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
Diff: https://reviews.apache.org/r/27210/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/
-----------------------------------------------------------
(Updated Oct. 26, 2014, 4:33 p.m.)
Review request for Ambari, Dmitro Lisnichenko and Mahadev Konar.
Bugs: AMBARI-7978
https://issues.apache.org/jira/browse/AMBARI-7978
Repository: ambari
Description
-------
When installing HDP, change the log directory of Falcon to non-default
for ex:
>From /var/log/falcon TO /grid/0/log/falcon
Install of falcon client fails and HDP cluster install fails with error that
/grid/0/log directory could not be found.
Expected:
log directory should be created if it does not exist
Log:
stderr:
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
stdout:
2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
2014-10-26 02:45:42,107 - Modifying group hadoop
2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
2014-10-26 02:45:42,229 - Modifying group nobody
2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
2014-10-26 02:45:42,318 - Modifying group users
2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
2014-10-26 02:45:42,423 - Modifying group nagios
2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
2014-10-26 02:45:42,518 - Modifying group knox
2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,616 - Modifying user hive
2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:42,730 - Modifying user oozie
2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
2014-10-26 02:45:42,830 - Modifying user nobody
2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,922 - Modifying user nagios
2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,014 - Modifying user ambari-qa
2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,116 - Modifying user flume
2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,205 - Modifying user hdfs
2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,290 - Modifying user knox
2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,379 - Modifying user storm
2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,470 - Modifying user mapred
2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,564 - Modifying user hbase
2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,659 - Modifying user tez
2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,753 - Modifying user zookeeper
2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,845 - Modifying user falcon
2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,935 - Modifying user sqoop
2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,022 - Modifying user yarn
2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,120 - Modifying user hcat
2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,483 - Package['unzip'] {}
2014-10-26 02:45:44,639 - Skipping installing existent package unzip
2014-10-26 02:45:44,640 - Package['curl'] {}
2014-10-26 02:45:44,734 - Skipping installing existent package curl
2014-10-26 02:45:44,735 - Package['hdp-select'] {}
2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
Diff: https://reviews.apache.org/r/27210/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/#review58561
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Oct. 26, 2014, 4:28 p.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27210/
> -----------------------------------------------------------
>
> (Updated Oct. 26, 2014, 4:28 p.m.)
>
>
> Review request for Ambari, Dmitro Lisnichenko and Dmytro Sen.
>
>
> Bugs: AMBARI-7978
> https://issues.apache.org/jira/browse/AMBARI-7978
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When installing HDP, change the log directory of Falcon to non-default
> for ex:
> From /var/log/falcon TO /grid/0/log/falcon
>
> Install of falcon client fails and HDP cluster install fails with error that
> /grid/0/log directory could not be found.
>
> Expected:
> log directory should be created if it does not exist
>
>
>
> Log:
> stderr:
> 2014-10-26 02:45:59,612 - Error while executing command 'install':
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
> self.configure(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
> falcon('client', action='config')
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
> owner=params.falcon_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
> raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
> Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
> stdout:
> 2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
> 2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
> 2014-10-26 02:45:42,107 - Modifying group hadoop
> 2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
> 2014-10-26 02:45:42,229 - Modifying group nobody
> 2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
> 2014-10-26 02:45:42,318 - Modifying group users
> 2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
> 2014-10-26 02:45:42,423 - Modifying group nagios
> 2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
> 2014-10-26 02:45:42,518 - Modifying group knox
> 2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:42,616 - Modifying user hive
> 2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:42,730 - Modifying user oozie
> 2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
> 2014-10-26 02:45:42,830 - Modifying user nobody
> 2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:42,922 - Modifying user nagios
> 2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:43,014 - Modifying user ambari-qa
> 2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,116 - Modifying user flume
> 2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,205 - Modifying user hdfs
> 2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,290 - Modifying user knox
> 2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,379 - Modifying user storm
> 2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,470 - Modifying user mapred
> 2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,564 - Modifying user hbase
> 2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
> 2014-10-26 02:45:43,659 - Modifying user tez
> 2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,753 - Modifying user zookeeper
> 2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,845 - Modifying user falcon
> 2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:43,935 - Modifying user sqoop
> 2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:44,022 - Modifying user yarn
> 2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
> 2014-10-26 02:45:44,120 - Modifying user hcat
> 2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
> 2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
> 2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
> 2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
> 2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
> 2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
> 2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
> 2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
> 2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
> 2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
> 2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
> 2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
> 2014-10-26 02:45:44,483 - Package['unzip'] {}
> 2014-10-26 02:45:44,639 - Skipping installing existent package unzip
> 2014-10-26 02:45:44,640 - Package['curl'] {}
> 2014-10-26 02:45:44,734 - Skipping installing existent package curl
> 2014-10-26 02:45:44,735 - Package['hdp-select'] {}
> 2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
> 2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
> 2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
> 2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
> 2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
> 2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
> 2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
> 2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
> 2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
> 2014-10-26 02:45:59,612 - Error while executing command 'install':
> Traceback (most recent call last):
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
> self.configure(env)
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
> falcon('client', action='config')
> File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
> owner=params.falcon_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
> raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
> Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
>
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
>
> Diff: https://reviews.apache.org/r/27210/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>
Re: Review Request 27210: Falcon client install fails and hence HDP
install fails when Falcon log directory is customized
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27210/
-----------------------------------------------------------
(Updated Oct. 26, 2014, 4:28 p.m.)
Review request for Ambari, Dmitro Lisnichenko and Dmytro Sen.
Bugs: AMBARI-7978
https://issues.apache.org/jira/browse/AMBARI-7978
Repository: ambari
Description
-------
When installing HDP, change the log directory of Falcon to non-default
for ex:
>From /var/log/falcon TO /grid/0/log/falcon
Install of falcon client fails and HDP cluster install fails with error that
/grid/0/log directory could not be found.
Expected:
log directory should be created if it does not exist
Log:
stderr:
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
stdout:
2014-10-26 02:45:42,034 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] {'environment': ..., 'not_if': 'test -e /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip', 'ignore_failures': True, 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:42,104 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//UnlimitedJCEPolicyJDK7.zip -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//UnlimitedJCEPolicyJDK7.zip'] due to not_if
2014-10-26 02:45:42,105 - Group['hadoop'] {'ignore_failures': False}
2014-10-26 02:45:42,107 - Modifying group hadoop
2014-10-26 02:45:42,228 - Group['nobody'] {'ignore_failures': False}
2014-10-26 02:45:42,229 - Modifying group nobody
2014-10-26 02:45:42,317 - Group['users'] {'ignore_failures': False}
2014-10-26 02:45:42,318 - Modifying group users
2014-10-26 02:45:42,422 - Group['nagios'] {'ignore_failures': False}
2014-10-26 02:45:42,423 - Modifying group nagios
2014-10-26 02:45:42,517 - Group['knox'] {'ignore_failures': False}
2014-10-26 02:45:42,518 - Modifying group knox
2014-10-26 02:45:42,615 - User['hive'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,616 - Modifying user hive
2014-10-26 02:45:42,729 - User['oozie'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:42,730 - Modifying user oozie
2014-10-26 02:45:42,829 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
2014-10-26 02:45:42,830 - Modifying user nobody
2014-10-26 02:45:42,921 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:42,922 - Modifying user nagios
2014-10-26 02:45:43,014 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,014 - Modifying user ambari-qa
2014-10-26 02:45:43,115 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,116 - Modifying user flume
2014-10-26 02:45:43,205 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,205 - Modifying user hdfs
2014-10-26 02:45:43,289 - User['knox'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,290 - Modifying user knox
2014-10-26 02:45:43,379 - User['storm'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,379 - Modifying user storm
2014-10-26 02:45:43,469 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,470 - Modifying user mapred
2014-10-26 02:45:43,563 - User['hbase'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,564 - Modifying user hbase
2014-10-26 02:45:43,658 - User['tez'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2014-10-26 02:45:43,659 - Modifying user tez
2014-10-26 02:45:43,752 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,753 - Modifying user zookeeper
2014-10-26 02:45:43,844 - User['falcon'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,845 - Modifying user falcon
2014-10-26 02:45:43,934 - User['sqoop'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:43,935 - Modifying user sqoop
2014-10-26 02:45:44,021 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,022 - Modifying user yarn
2014-10-26 02:45:44,119 - User['hcat'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2014-10-26 02:45:44,120 - Modifying user hcat
2014-10-26 02:45:44,223 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,225 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2014-10-26 02:45:44,289 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2014-10-26 02:45:44,290 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2014-10-26 02:45:44,291 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] {'not_if': 'test $(id -u hbase) -gt 1000'}
2014-10-26 02:45:44,354 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/grid/0/hadoop/hbase 2>/dev/null'] due to not_if
2014-10-26 02:45:44,355 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
2014-10-26 02:45:44,356 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2014-10-26 02:45:44,419 - Skipping Link['/etc/hadoop/conf'] due to not_if
2014-10-26 02:45:44,440 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2014-10-26 02:45:44,463 - Repository['HDP-2.2'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos5/2.x/BUILDS/2.2.0.0-1052', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP', 'mirror_list': None}
2014-10-26 02:45:44,476 - File['/etc/yum.repos.d/HDP.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,477 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': 'repo_suse_rhel.j2', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2014-10-26 02:45:44,482 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': Template('repo_suse_rhel.j2')}
2014-10-26 02:45:44,483 - Package['unzip'] {}
2014-10-26 02:45:44,639 - Skipping installing existent package unzip
2014-10-26 02:45:44,640 - Package['curl'] {}
2014-10-26 02:45:44,734 - Skipping installing existent package curl
2014-10-26 02:45:44,735 - Package['hdp-select'] {}
2014-10-26 02:45:44,822 - Skipping installing existent package hdp-select
2014-10-26 02:45:44,824 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] {'environment': ..., 'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,884 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/AMBARI-artifacts/ ; curl -kf -x "" --retry 10 http://ip-172-31-4-239.ec2.internal:8080/resources//jdk-7u67-linux-x64.tar.gz -o /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz'] due to not_if
2014-10-26 02:45:44,886 - Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] {'not_if': 'test -e /usr/jdk64/jdk1.7.0_67/bin/java', 'path': ['/bin', '/usr/bin/']}
2014-10-26 02:45:44,945 - Skipping Execute['mkdir -p /usr/jdk64 ; cd /usr/jdk64 ; tar -xf /var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jdk-7u67-linux-x64.tar.gz > /dev/null 2>&1'] due to not_if
2014-10-26 02:45:45,249 - Package['falcon_2_2_*'] {}
2014-10-26 02:45:45,344 - Installing package falcon_2_2_* ('/usr/bin/yum -d 0 -e 0 -y install falcon_2_2_*')
2014-10-26 02:45:59,601 - Directory['/var/run/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Directory['/grid/0/log/falcon'] {'owner': 'falcon'}
2014-10-26 02:45:59,612 - Creating directory Directory['/grid/0/log/falcon']
2014-10-26 02:45:59,612 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 122, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 26, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon_client.py", line 32, in configure
falcon('client', action='config')
File "/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py", line 29, in falcon
owner=params.falcon_user
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 152, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
Fail: Applying Directory['/grid/0/log/falcon'] failed, parent directory /grid/0/log doesn't exist
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/package/scripts/falcon.py 05a685a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py 325c06a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 57bf715
Diff: https://reviews.apache.org/r/27210/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk