You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2017/12/20 14:33:37 UTC
Review Request 64751: Fix Broken Symlinks on Stack Distribution
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley and Nate Cole.
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
-------
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b9
Diff: https://reviews.apache.org/r/64751/diff/1/
Testing
-------
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 20, 2017, 5:03 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
> > Lines 199-201 (patched)
> > <https://reviews.apache.org/r/64751/diff/1/?file=1925645#file1925645line200>
> >
> > I'd say just return earlier like this:
> >
> > if not stack_version:
> > Logger.warn("Unable to fix {0} since there is no known installed version for this component".format(package_name))
Stack is being checked for every package in a loop, so return here would break the execution flow
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194249
-----------------------------------------------------------
On Dec. 20, 2017, 4:33 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2017, 4:33 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There are two scenarios to cover here:
>
> # Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
> # The conf pointers of a component are broken
>
> In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
>
> The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
>
> Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
>
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
>
> Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
> package, directory_struct['conf_dir']))
>
> return
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b9
>
>
> Diff: https://reviews.apache.org/r/64751/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
>
> Live cluster check
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194249
-----------------------------------------------------------
Fix it, then Ship it!
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
Lines 179 (patched)
<https://reviews.apache.org/r/64751/#comment272976>
nit: Attempting to fix any configuration symlinks which are not in the correct state
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
Lines 199-201 (patched)
<https://reviews.apache.org/r/64751/#comment272977>
I'd say just return earlier like this:
if not stack_version:
Logger.warn("Unable to fix {0} since there is no known installed version for this component".format(package_name))
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json
Lines 1075-1080 (original), 1090-1096 (patched)
<https://reviews.apache.org/r/64751/#comment272978>
nifi is supported by hdp-select?
- Jonathan Hurley
On Dec. 20, 2017, 9:33 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2017, 9:33 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There are two scenarios to cover here:
>
> # Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
> # The conf pointers of a component are broken
>
> In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
>
> The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
>
> Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
>
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
>
> Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
> package, directory_struct['conf_dir']))
>
> return
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b9
>
>
> Diff: https://reviews.apache.org/r/64751/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
>
> Live cluster check
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194333
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On Dec. 21, 2017, 8:25 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2017, 8:25 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There are two scenarios to cover here:
>
> # Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
> # The conf pointers of a component are broken
>
> In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
>
> The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
>
> Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
>
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
>
> Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
> package, directory_struct['conf_dir']))
>
> return
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd2cb
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64751/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
> Live cluster check
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
-----------------------------------------------------------
(Updated Dec. 26, 2017, 7:10 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
-------
added Livy2
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
-------
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py add5a7789e
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py 0180a315b6
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 5cff342076
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json dc71b4da4f
Diff: https://reviews.apache.org/r/64751/diff/3/
Changes: https://reviews.apache.org/r/64751/diff/2-3/
Testing
-------
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/#review194353
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Dec. 21, 2017, 8:25 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64751/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2017, 8:25 a.m.)
>
>
> Review request for Ambari, Jonathan Hurley and Nate Cole.
>
>
> Bugs: AMBARI-22678
> https://issues.apache.org/jira/browse/AMBARI-22678
>
>
> Repository: ambari
>
>
> Description
> -------
>
> There are two scenarios to cover here:
>
> # Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
> # The conf pointers of a component are broken
>
> In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
> {code}
> /etc/component/conf (directory)
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> {code}
>
> The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
> {code}
> /etc/component/conf -> /usr/hdp/current/component
> /usr/hdp/current/component -> /usr/hdp/v1/component
> /usr/hdp/v1/component -> /etc/component/conf
> /usr/hdp/v2/component -> /etc/component/v2/0
> {code}
>
> Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
>
> Most likely the proper fix will be:
> - Iterate over the entire known conf-select structure
> - Check to see the state /etc/component/conf - if it's bad, fix it to defaults
>
> Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
> {code}
> stack_name = Script.get_stack_name()
> for directory_struct in dirs:
> if not os.path.exists(directory_struct['conf_dir']):
> Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
> package, directory_struct['conf_dir']))
>
> return
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd2cb
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64751/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
> Live cluster check
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64751: Fix Broken Symlinks on Stack Distribution
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64751/
-----------------------------------------------------------
(Updated Dec. 21, 2017, 3:25 p.m.)
Review request for Ambari, Jonathan Hurley and Nate Cole.
Changes
-------
Addressed comments
Bugs: AMBARI-22678
https://issues.apache.org/jira/browse/AMBARI-22678
Repository: ambari
Description
-------
There are two scenarios to cover here:
# Ambari never conf-select'd a component (maybe because of a bug or because the component didn't support it)
# The conf pointers of a component are broken
In either event, when distributing a new stack, the code detects this problem (as it would on a first-time install) and tries to fix it:
{code}
/etc/component/conf (directory)
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
{code}
The stack distribution thinks this is a first-time installed and tries to fix the symlinks. We end up with:
{code}
/etc/component/conf -> /usr/hdp/current/component
/usr/hdp/current/component -> /usr/hdp/v1/component
/usr/hdp/v1/component -> /etc/component/conf
/usr/hdp/v2/component -> /etc/component/v2/0
{code}
Because we're only conf-selecting v2, v1 never gets corrected since it's already installed. Thus, we have a circular symlink.
Most likely the proper fix will be:
- Iterate over the entire known conf-select structure
- Check to see the state /etc/component/conf - if it's bad, fix it to defaults
Chances are we can do this directly in {{conf_select.convert_conf_directories_to_symlinks}}:
{code}
stack_name = Script.get_stack_name()
for directory_struct in dirs:
if not os.path.exists(directory_struct['conf_dir']):
Logger.info("Skipping the conf-select tool on {0} since {1} does not exist.".format(
package, directory_struct['conf_dir']))
return
{code}
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py c8497cd2cb
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
Diff: https://reviews.apache.org/r/64751/diff/2/
Changes: https://reviews.apache.org/r/64751/diff/1-2/
Testing
-------
mvn clean test
Live cluster check
Thanks,
Dmitro Lisnichenko