You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/12/11 23:45:03 UTC
Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker restart
failed
on downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 7d90dca
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
PRELIMINARY code review. Still need to test and fix unit tests.
TODO:
1. Fresh install
2. EU from 2.1 to 2.3
3. RU from 2.2 to 2.3, and downgrade
4. RU/EU from 2.2 to 2.2+
5. RU/EU from 2.3 to 2.3+
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110142
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 295)
<https://reviews.apache.org/r/41273/#comment169910>
Please review this logic since it matters during RU downgrade and EU.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 460)
<https://reviews.apache.org/r/41273/#comment169912>
Before, on a host without hadoop components, it would fail on the "select" call and not execute the rest of the code. Now, the /etc/hadoop/conf symlink is pointed to the /etc/hadoop/conf.backup folder
ambari-common/src/main/python/resource_management/libraries/script/script.py (line 304)
<https://reviews.apache.org/r/41273/#comment169909>
Called by conf_select.py to determine which hadoop conf dir to use.
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py (line 54)
<https://reviews.apache.org/r/41273/#comment169913>
This was a bug in Kafka.
- Alejandro Fernandez
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Dec. 14, 2015, 12:13 a.m., Jayush Luniya wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, line 350
> > <https://reviews.apache.org/r/41273/diff/4/?file=1161950#file1161950line350>
> >
> > since hdp-select has not yet been called, if HDP 2.2 -> HDP 2.3 upgrade is in progress, we would end up creating a symlink to /usr/hdp/2.2.x/hadoop/conf?
This just returns which symlink to use at this time, which can potentially also be during a downgrade.
It's not calling conf-select, so I think it's safe.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110146
-----------------------------------------------------------
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Dec. 14, 2015, 12:13 a.m., Jayush Luniya wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py, line 476
> > <https://reviews.apache.org/r/41273/diff/4/?file=1161950#file1161950line476>
> >
> > else
Else block would mean it got an invalid param type, I'll log it.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110146
-----------------------------------------------------------
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110146
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 349)
<https://reviews.apache.org/r/41273/#comment169915>
since hdp-select has not yet been called, if HDP 2.2 -> HDP 2.3 upgrade is in progress, we would end up creating a symlink to /usr/hdp/2.2.x/hadoop/conf?
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 475)
<https://reviews.apache.org/r/41273/#comment169916>
else
- Jayush Luniya
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Dec. 14, 2015, 3:41 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py, lines 135-136
> > <https://reviews.apache.org/r/41273/diff/4/?file=1161954#file1161954line135>
> >
> > I think this is still a major bug. If for some reason the conf.backup directory doesn't exist, we won't change the symlink point of /etc/<component>/conf away from /usr/hdp/current/<component>/conf - on a 2.2 downgrade, this will lead to a circular reference.
> >
> > Even if the backup directory doesn't exist, we must unlink /etc/<component>/conf and make it a folder again.
Good call, will fix.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110159
-----------------------------------------------------------
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110159
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py (lines 135 - 136)
<https://reviews.apache.org/r/41273/#comment169926>
I think this is still a major bug. If for some reason the conf.backup directory doesn't exist, we won't change the symlink point of /etc/<component>/conf away from /usr/hdp/current/<component>/conf - on a 2.2 downgrade, this will lead to a circular reference.
Even if the backup directory doesn't exist, we must unlink /etc/<component>/conf and make it a folder again.
- Jonathan Hurley
On Dec. 13, 2015, 5:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 5:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110150
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 349)
<https://reviews.apache.org/r/41273/#comment169919>
This is legacy code, I don't want to muck around with it too much until we do a complete refactor.
- Alejandro Fernandez
On Dec. 13, 2015, 10:58 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 10:58 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Still need to fix unit tests.
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110240
-----------------------------------------------------------
Ship it!
Looks good
- Dmitro Lisnichenko
On Dec. 14, 2015, 9:17 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 9:17 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Dec. 14, 2015, 6:52 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py, lines 188-190
> > <https://reviews.apache.org/r/41273/diff/6/?file=1162113#file1162113line188>
> >
> > Just found this bug and thought you should fix it as part of your review.
> >
> > We should _only_ being doing the symlinking if coming from < 2.3 to >= 2.3
> >
> > Consider this case. You install 2.3.0.0 and all links are correct. /etc/<component>/conf is correctly a pointer to /usr/hdp/current/<component>/conf
> >
> > But then you distribute 2.3.4.0 in preparation for an upgrade. This code will unset the above link and put it back to /etc/<component>/conf.backup
> >
> > Basically we need to add 1 more if-statement here that does this:
> >
> > if existing-current-version >= HDP 2.3:
> > Logger.info("Configuration symlinks are already correct on {0}".format(stack_version))
> > return
Actually, this is already handled in install_packges.py
# Version installing must be 2.3.0.0 or higher
if version.compare_versions(version.format_hdp_stack_version(args[1]), "2.3.0.0") < 0:
Logger.info("Configuration symlinks are not needed for {0}, only HDP-2.3+".format(stack_version))
return
# Current HDP stack must be less than 2.3
# if already on HDP 2.3, then there's nothing to do in terms of linking configs
if self.current_hdp_stack_version and compare_versions(self.current_hdp_stack_version, '2.3') >= 0:
Logger.info("The current cluster stack of {0} does not require linking configurations".format(stack_version))
return
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110286
-----------------------------------------------------------
On Dec. 14, 2015, 7:17 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 7:17 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110286
-----------------------------------------------------------
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (lines 188 - 190)
<https://reviews.apache.org/r/41273/#comment170100>
Just found this bug and thought you should fix it as part of your review.
We should _only_ being doing the symlinking if coming from < 2.3 to >= 2.3
Consider this case. You install 2.3.0.0 and all links are correct. /etc/<component>/conf is correctly a pointer to /usr/hdp/current/<component>/conf
But then you distribute 2.3.4.0 in preparation for an upgrade. This code will unset the above link and put it back to /etc/<component>/conf.backup
Basically we need to add 1 more if-statement here that does this:
if existing-current-version >= HDP 2.3:
Logger.info("Configuration symlinks are already correct on {0}".format(stack_version))
return
- Jonathan Hurley
On Dec. 14, 2015, 2:17 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 2:17 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110248
-----------------------------------------------------------
Ship it!
Jonathan's comment is important, mine are cosmetic.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 364)
<https://reviews.apache.org/r/41273/#comment170053>
Remove 'pass' here
ambari-common/src/main/python/resource_management/libraries/script/script.py (line 306)
<https://reviews.apache.org/r/41273/#comment170049>
Could probably format this a little for readability. Use default()?
- Nate Cole
On Dec. 14, 2015, 2:17 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 2:17 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110217
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (lines 302 - 306)
<https://reviews.apache.org/r/41273/#comment170010>
If the stack is 2.3, then the first if-statement will be true - the 2nd if-statement will never be entered.
- Jonathan Hurley
On Dec. 14, 2015, 2:17 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 2:17 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 10:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py, lines 192-195
> > <https://reviews.apache.org/r/41273/diff/6-7/?file=1162113#file1162113line192>
> >
> > +1! Glad this is back. It's needed.
That breaks AMBARI-14293 back.
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110298
-----------------------------------------------------------
On Dec. 14, 2015, 10 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 10 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py c9c70de
> ambari-common/src/main/python/resource_management/libraries/script/script.py 9dc402d
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 470661a
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py 885d650
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 2c39614
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 3260715
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 75aa4a1
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 7edd867
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 48d55b1
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 490b3bf
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 1c782bd
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110298
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py (lines 192 - 195)
<https://reviews.apache.org/r/41273/#comment170114>
+1! Glad this is back. It's needed.
- Jonathan Hurley
On Dec. 14, 2015, 3 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 3 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py c9c70de
> ambari-common/src/main/python/resource_management/libraries/script/script.py 9dc402d
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 470661a
> ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py 885d650
> ambari-server/src/test/python/custom_actions/test_ru_set_all.py 2c39614
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 3260715
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 75aa4a1
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 7edd867
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 48d55b1
> ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 490b3bf
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 1c782bd
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> Tested on the following code paths,
>
> 1. Fresh install of HDP 2.1
> 2. Installed bits for HDP 2.3 and verified the conf folder structure
> 3. EU to HDP 2.3
> 4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
>
>
> 1. Fresh install of HDP 2.2
> 2. Installed bits for HDP 2.3 and verfied the conf folder structure
> 3. RU to HDP 2.3 and downgrade
>
> Python unit tests passed,
> ----------------------------------------------------------------------
> Total run:853
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 8 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py c9c70de
ambari-common/src/main/python/resource_management/libraries/script/script.py 9dc402d
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 470661a
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py 885d650
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 2c39614
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 3260715
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 75aa4a1
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 7edd867
ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 48d55b1
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 490b3bf
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 1c782bd
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
Tested on the following code paths,
1. Fresh install of HDP 2.1
2. Installed bits for HDP 2.3 and verified the conf folder structure
3. EU to HDP 2.3
4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
1. Fresh install of HDP 2.2
2. Installed bits for HDP 2.3 and verfied the conf folder structure
3. RU to HDP 2.3 and downgrade
Python unit tests passed,
----------------------------------------------------------------------
Total run:853
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 7:17 a.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
Tested on the following code paths,
1. Fresh install of HDP 2.1
2. Installed bits for HDP 2.3 and verified the conf folder structure
3. EU to HDP 2.3
4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
1. Fresh install of HDP 2.2
2. Installed bits for HDP 2.3 and verfied the conf folder structure
3. RU to HDP 2.3 and downgrade
Python unit tests passed,
----------------------------------------------------------------------
Total run:853
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 41273: AMBARI-14307. RU: Kafka broker restart
failed on
downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:25 a.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
Addressed comments and fixed unit tests.
Summary (updated)
-----------------
AMBARI-14307. RU: Kafka broker restart failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of /etc/hadoop/conf
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 186b847
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py d2ffb3c
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 9e4560c
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py 5fd5ef6
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41273/diff/
Testing (updated)
-------
Tested on the following code paths,
1. Fresh install of HDP 2.1
2. Installed bits for HDP 2.3 and verified the conf folder structure
3. EU to HDP 2.3
4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
1. Fresh install of HDP 2.2
2. Installed bits for HDP 2.3 and verfied the conf folder structure
3. RU to HDP 2.3 and downgrade
Python unit tests passed,
----------------------------------------------------------------------
Total run:853
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 13, 2015, 10:58 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description (updated)
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka (this host does not contain any hadoop components)
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
Diff: https://reviews.apache.org/r/41273/diff/
Testing (updated)
-------
Tested on the following code paths,
1. Fresh install of HDP 2.1
2. Installed bits for HDP 2.3 and verified the conf folder structure
3. EU to HDP 2.3
4. Installed HDFS client on the host that didn't have /etc/hadoop/conf and then it symlinked it to /usr/hdp/current/hadoop-client/conf
1. Fresh install of HDP 2.2
2. Installed bits for HDP 2.3 and verfied the conf folder structure
3. RU to HDP 2.3 and downgrade
Still need to fix unit tests.
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 13, 2015, 10:50 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
Got it working. Still need to fix unit tests.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
PRELIMINARY code review. Still need to test and fix unit tests.
TODO:
1. Fresh install
2. EU from 2.1 to 2.3
3. RU from 2.2 to 2.3, and downgrade
4. RU/EU from 2.2 to 2.2+
5. RU/EU from 2.3 to 2.3+
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 12, 2015, 8:44 a.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
More fixes, still doesn't work for EU from 2.1 to 2.3
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 7d90dca
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py f8b38db
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
PRELIMINARY code review. Still need to test and fix unit tests.
TODO:
1. Fresh install
2. EU from 2.1 to 2.3
3. RU from 2.2 to 2.3, and downgrade
4. RU/EU from 2.2 to 2.2+
5. RU/EU from 2.3 to 2.3+
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/
-----------------------------------------------------------
(Updated Dec. 12, 2015, 1:37 a.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14307
https://issues.apache.org/jira/browse/AMBARI-14307
Repository: ambari
Description
-------
* Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
* HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka
* RU from to 2.3.4.0 and reach the Finalize screen.
* Then downgrade,
Kafka broker restart failed on downgrade
```
Traceback (most recent call last):
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
BeforeStartHook().execute()
File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
method(env)
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
create_topology_script_and_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
create_topology_mapping()
File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
group=params.user_group)
File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
self.env.run()
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
self.run_action(resource, action)
File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
provider_action()
File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
```
This failed because /etc/hadoop/conf is now a circular symlink
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 7d90dca
ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
Diff: https://reviews.apache.org/r/41273/diff/
Testing
-------
PRELIMINARY code review. Still need to test and fix unit tests.
TODO:
1. Fresh install
2. EU from 2.1 to 2.3
3. RU from 2.2 to 2.3, and downgrade
4. RU/EU from 2.2 to 2.2+
5. RU/EU from 2.3 to 2.3+
Thanks,
Alejandro Fernandez
Re: Review Request 41273: [DRAFT] AMBARI-14307. RU: Kafka broker
restart
failed on downgrade from HDP 2.3 to 2.2 due to circular symlink of
/etc/hadoop/conf
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41273/#review110033
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 320)
<https://reviews.apache.org/r/41273/#comment169784>
This is the significant change in this review.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 393)
<https://reviews.apache.org/r/41273/#comment169785>
Moved this from install_packages.py to here since it looks cleaner.
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 525)
<https://reviews.apache.org/r/41273/#comment169786>
I wanted to explicitly use a string, either backup or current, to indicate where the symlink is going to.
ambari-common/src/main/python/resource_management/libraries/script/script.py (line 272)
<https://reviews.apache.org/r/41273/#comment169787>
This is called by conf_select.py
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
<https://reviews.apache.org/r/41273/#comment169789>
Moved this to conf_select.py
- Alejandro Fernandez
On Dec. 11, 2015, 10:45 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41273/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2015, 10:45 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14307
> https://issues.apache.org/jira/browse/AMBARI-14307
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Ambari 2.2 at hash for AMBARI-14237 (e15d1e0fb3777f79cb473cf5de0f7a6bf30e63bd)
> * HDP 2.2.8.0 with HDFS, YARN/MR, ZK, Kafka
> * RU from to 2.3.4.0 and reach the Finalize screen.
> * Then downgrade,
>
> Kafka broker restart failed on downgrade
> ```
> Traceback (most recent call last):
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 39, in <module>
> BeforeStartHook().execute()
> File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 217, in execute
> method(env)
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py\", line 36, in hook
> create_topology_script_and_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 69, in create_topology_script_and_mapping
> create_topology_mapping()
> File \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py\", line 36, in create_topology_mapping
> group=params.user_group)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 154, in __init__
> self.env.run()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 158, in run
> self.run_action(resource, action)
> File \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", line 121, in run_action
> provider_action()
> File \"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\", line 144, in action_create
> raise Fail(\"Applying %s failed, looped symbolic links found while resolving %s\" % (self.resource, path))
> resource_management.core.exceptions.Fail: Applying Directory['/etc/hadoop/conf'] failed, looped symbolic links found while resolving /etc/hadoop/conf
> Error: Error: Unable to run the custom hook script ['/usr/bin/python2', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py', 'START', '/var/lib/ambari-agent/data/command-2873.json', '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', '/var/lib/ambari-agent/data/structured-out-2873.json', 'INFO', '/var/lib/ambari-agent/tmp']
> ```
>
>
> This failed because /etc/hadoop/conf is now a circular symlink
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 7fd64b5
> ambari-common/src/main/python/resource_management/libraries/functions/hdp_select.py 7d90dca
> ambari-common/src/main/python/resource_management/libraries/script/script.py 398cee8
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 85d0d25
>
> Diff: https://reviews.apache.org/r/41273/diff/
>
>
> Testing
> -------
>
> PRELIMINARY code review. Still need to test and fix unit tests.
>
> TODO:
> 1. Fresh install
> 2. EU from 2.1 to 2.3
> 3. RU from 2.2 to 2.3, and downgrade
> 4. RU/EU from 2.2 to 2.2+
> 5. RU/EU from 2.3 to 2.3+
>
>
> Thanks,
>
> Alejandro Fernandez
>
>