You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/09/21 07:16:17 UTC

Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38543/
-----------------------------------------------------------

Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.

Bugs: AMBARI-13164
https://issues.apache.org/jira/browse/AMBARI-13164

Repository: ambari

Description
-------

Starting in HDP 2.3.2.0, Knox will version its data directory.

HDP 2.3.2.0 is meant to support Debian & Ubuntu.
A customer installing HDP 2.3.2.0 would receive the following for knox:
/usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
{version}

Hence, all of the data directories in /var/lib/knox/ would be versioned. This is because the Debian installer fails if it tries to create the /var/lib/knox/data folder and it already exists (since it belongs to a previous version); further, you would get data loss if you tried to uninstall the newer version, since it would wipe out the folder that was used by the previous version.

So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.

This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.

There are 3 scenarios:

1. Ambari 2.1.2 with HDP 2.3.0.0:
Ambari must continue to use the non-versioned data directory.

2. Ambari 2.1.2 with HDP 2.3.2.0:
A brand new install will use the versioned data directory out-of-the-box.

3. Ambari 2.1.0 with HDP 2.3.0.0:
Technically, this allows registering bits for HDP 2.3.2.0 and performing an RU to it. However, because HDP 2.3.2.0 uses a versioned data dir, it means that any customer with Knox that plans to perform an RU must first upgrade Ambari to 2.1.2, which is equivalent to starting with case #1.

Because RU allows the following paths,
2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)

It means that it must always copy any contents from
previous_data_dir to /usr/hdp/{new-version}/knox/data

In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data

Diffs
-----

ambari-common/src/main/python/resource_management/libraries/script/script.py e0a7877
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 990fc87
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/upgrade.py 2c805fa
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager_upgrade.py 9ac2d63
ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 3ae42a0

Diff: https://reviews.apache.org/r/38543/diff/

Testing
-------

Ran python unit tests, which passed.
Verified during RU in several scenarios.
1. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Before Finalizing, Downgraded.
2. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Finalized.
3. Steps from #2, and then perform an RU to a higher build in HDP 2.3.2.0

Thanks,

Alejandro Fernandez

Re: Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

Posted by Jonathan Hurley <jh...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38543/#review99819
-----------------------------------------------------------

Ship it!


Ship It!

- Jonathan Hurley


On Sept. 21, 2015, 4:18 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38543/
> -----------------------------------------------------------
> 
> (Updated Sept. 21, 2015, 4:18 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.
> 
> 
> Bugs: AMBARI-13164
>     https://issues.apache.org/jira/browse/AMBARI-13164
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting in HDP 2.3.2.0, Knox will version its data directory.
> 
> HDP 2.3.2.0 is meant to support Debian & Ubuntu.
> A customer installing HDP 2.3.2.0 would receive the following for knox:
> /usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
> {version}
> 
> Hence, all of the data directories in /var/lib/knox/ would be versioned. This is because the Debian installer fails if it tries to create the /var/lib/knox/data folder and it already exists (since it belongs to a previous version); further, you would get data loss if you tried to uninstall the newer version, since it would wipe out the folder that was used by the previous version.
> 
> So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.
> 
> This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.
> 
> There are 3 scenarios:
> 
> 1. Ambari 2.1.2 with HDP 2.3.0.0:
> Ambari must continue to use the non-versioned data directory.
> 
> 2. Ambari 2.1.2 with HDP 2.3.2.0:
> A brand new install will use the versioned data directory out-of-the-box.
> 
> 3. Ambari 2.1.0 with HDP 2.3.0.0:
> Technically, this allows registering bits for HDP 2.3.2.0 and performing an RU to it. However, because HDP 2.3.2.0 uses a versioned data dir, it means that any customer with Knox that plans to perform an RU must first upgrade Ambari to 2.1.2, which is equivalent to starting with case #1.
> 
> Because RU allows the following paths, 
> 2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
> 2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)
> 
> It means that it must always copy any contents from 
> previous_data_dir to  /usr/hdp/{new-version}/knox/data
> 
> In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
> In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/script/script.py e0a7877 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 990fc87 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/upgrade.py 2c805fa 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager_upgrade.py 9ac2d63 
>   ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 3ae42a0 
> 
> Diff: https://reviews.apache.org/r/38543/diff/
> 
> 
> Testing
> -------
> 
> Ran python unit tests, which passed.
> Verified during RU in several scenarios.
> 1. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Before Finalizing, Downgraded.
> 2. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Finalized.
> 3. Steps from #2, and then perform an RU to a higher build in HDP 2.3.2.0
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

Posted by Alejandro Fernandez <af...@hortonworks.com>.

(Updated Sept. 21, 2015, 8:39 p.m.)

Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.

Changes
-------

Addressed comments, unit tests passed.

Bugs: AMBARI-13164
https://issues.apache.org/jira/browse/AMBARI-13164

Repository: ambari

Description
-------

Starting in HDP 2.3.2.0, Knox will version its data directory.

HDP 2.3.2.0 is meant to support Debian & Ubuntu.
A customer installing HDP 2.3.2.0 would receive the following for knox:
/usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
{version}

So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.

This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.

There are 3 scenarios:

1. Ambari 2.1.2 with HDP 2.3.0.0:
Ambari must continue to use the non-versioned data directory.

2. Ambari 2.1.2 with HDP 2.3.2.0:
A brand new install will use the versioned data directory out-of-the-box.

Because RU allows the following paths,
2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)

It means that it must always copy any contents from
previous_data_dir to /usr/hdp/{new-version}/knox/data

In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/38543/diff/

Testing
-------

Thanks,

Alejandro Fernandez

Re: Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

Posted by Jonathan Hurley <jh...@hortonworks.com>.


> On Sept. 21, 2015, 4:24 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py, line 53
> > <https://reviews.apache.org/r/38543/diff/2/?file=1078489#file1078489line53>
> >
> >     Make this different name than your username :)

Too funny.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38543/#review99818
-----------------------------------------------------------


On Sept. 21, 2015, 4:18 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38543/
> -----------------------------------------------------------
> 
> (Updated Sept. 21, 2015, 4:18 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.
> 
> 
> Bugs: AMBARI-13164
>     https://issues.apache.org/jira/browse/AMBARI-13164
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting in HDP 2.3.2.0, Knox will version its data directory.
> 
> HDP 2.3.2.0 is meant to support Debian & Ubuntu.
> A customer installing HDP 2.3.2.0 would receive the following for knox:
> /usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
> {version}
> 
> Hence, all of the data directories in /var/lib/knox/ would be versioned. This is because the Debian installer fails if it tries to create the /var/lib/knox/data folder and it already exists (since it belongs to a previous version); further, you would get data loss if you tried to uninstall the newer version, since it would wipe out the folder that was used by the previous version.
> 
> So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.
> 
> This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.
> 
> There are 3 scenarios:
> 
> 1. Ambari 2.1.2 with HDP 2.3.0.0:
> Ambari must continue to use the non-versioned data directory.
> 
> 2. Ambari 2.1.2 with HDP 2.3.2.0:
> A brand new install will use the versioned data directory out-of-the-box.
> 
> 3. Ambari 2.1.0 with HDP 2.3.0.0:
> Technically, this allows registering bits for HDP 2.3.2.0 and performing an RU to it. However, because HDP 2.3.2.0 uses a versioned data dir, it means that any customer with Knox that plans to perform an RU must first upgrade Ambari to 2.1.2, which is equivalent to starting with case #1.
> 
> Because RU allows the following paths, 
> 2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
> 2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)
> 
> It means that it must always copy any contents from 
> previous_data_dir to  /usr/hdp/{new-version}/knox/data
> 
> In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
> In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/script/script.py e0a7877 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 990fc87 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/upgrade.py 2c805fa 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager_upgrade.py 9ac2d63 
>   ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 3ae42a0 
> 
> Diff: https://reviews.apache.org/r/38543/diff/
> 
> 
> Testing
> -------
> 
> Ran python unit tests, which passed.
> Verified during RU in several scenarios.
> 1. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Before Finalizing, Downgraded.
> 2. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Finalized.
> 3. Steps from #2, and then perform an RU to a higher build in HDP 2.3.2.0
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

Posted by Nate Cole <nc...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38543/#review99818
-----------------------------------------------------------

Ship it!



ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py (line 53)
<https://reviews.apache.org/r/38543/#comment156794>

    Make this different name than your username :)


- Nate Cole


On Sept. 21, 2015, 4:18 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38543/
> -----------------------------------------------------------
> 
> (Updated Sept. 21, 2015, 4:18 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.
> 
> 
> Bugs: AMBARI-13164
>     https://issues.apache.org/jira/browse/AMBARI-13164
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting in HDP 2.3.2.0, Knox will version its data directory.
> 
> HDP 2.3.2.0 is meant to support Debian & Ubuntu.
> A customer installing HDP 2.3.2.0 would receive the following for knox:
> /usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
> {version}
> 
> Hence, all of the data directories in /var/lib/knox/ would be versioned. This is because the Debian installer fails if it tries to create the /var/lib/knox/data folder and it already exists (since it belongs to a previous version); further, you would get data loss if you tried to uninstall the newer version, since it would wipe out the folder that was used by the previous version.
> 
> So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.
> 
> This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.
> 
> There are 3 scenarios:
> 
> 1. Ambari 2.1.2 with HDP 2.3.0.0:
> Ambari must continue to use the non-versioned data directory.
> 
> 2. Ambari 2.1.2 with HDP 2.3.2.0:
> A brand new install will use the versioned data directory out-of-the-box.
> 
> 3. Ambari 2.1.0 with HDP 2.3.0.0:
> Technically, this allows registering bits for HDP 2.3.2.0 and performing an RU to it. However, because HDP 2.3.2.0 uses a versioned data dir, it means that any customer with Knox that plans to perform an RU must first upgrade Ambari to 2.1.2, which is equivalent to starting with case #1.
> 
> Because RU allows the following paths, 
> 2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
> 2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)
> 
> It means that it must always copy any contents from 
> previous_data_dir to  /usr/hdp/{new-version}/knox/data
> 
> In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
> In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/script/script.py e0a7877 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 990fc87 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/upgrade.py 2c805fa 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager_upgrade.py 9ac2d63 
>   ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 3ae42a0 
> 
> Diff: https://reviews.apache.org/r/38543/diff/
> 
> 
> Testing
> -------
> 
> Ran python unit tests, which passed.
> Verified during RU in several scenarios.
> 1. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Before Finalizing, Downgraded.
> 2. Fresh install of HDP 2.3.0.0 and RU to 2.3.2.0 with Knox. Finalized.
> 3. Steps from #2, and then perform an RU to a higher build in HDP 2.3.2.0
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 38543: RU: Knox to use versioned data dir starting in HDP 2.3.2.0

Posted by Alejandro Fernandez <af...@hortonworks.com>.

(Updated Sept. 21, 2015, 8:18 p.m.)

Review request for Ambari, Jonathan Hurley, Jayush Luniya, Nate Cole, Srimanth Gunturi, and Sid Wagle.

Bugs: AMBARI-13164
https://issues.apache.org/jira/browse/AMBARI-13164

Repository: ambari

Description
-------

Starting in HDP 2.3.2.0, Knox will version its data directory.

HDP 2.3.2.0 is meant to support Debian & Ubuntu.
A customer installing HDP 2.3.2.0 would receive the following for knox:
/usr/hdp/current/knox/data is a symlink to /var/lib/knox/data_$
{version}

So the problem doesn't arise on the initial install, but rather on installing an additional version for Rolling Upgrade.

This means that any customer planning to upgrade Knox must first upgrade Ambari to 2.1.2.

There are 3 scenarios:

1. Ambari 2.1.2 with HDP 2.3.0.0:
Ambari must continue to use the non-versioned data directory.

2. Ambari 2.1.2 with HDP 2.3.2.0:
A brand new install will use the versioned data directory out-of-the-box.

Because RU allows the following paths,
2.3.0.0 (non-versioned data dir) -> 2.3.2.0 (versioned data dir)
2.3.2.0 (versioned data dir) -> 2.3.3.- (versioned data dir, once it is available)

It means that it must always copy any contents from
previous_data_dir to /usr/hdp/{new-version}/knox/data

In the case of 2.3.0.0 -> 2.3.+, previous_data_dir is /var/lib/knox/data/
In the case of 2.3.+ -> 2.3.+, previous_data_dir is /usr/hdp/{prev-version}/knox/data

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/38543/diff/

Testing
-------

Thanks,

Alejandro Fernandez