You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/11/08 21:22:33 UTC
Review Request 63684: Patch Upgrades Broken For Clients Due To
Versioned LD Library
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22386
https://issues.apache.org/jira/browse/AMBARI-22386
Repository: ambari
Description
-------
When services are patched to different versions, jobs being run in containers are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in via the job.
Consider the following situation:
- Base cluster installed at 2.6.3.0-1
- Enable some sort of codec, like snappy
- YARN patched to 2.6.3.1-1
- Hive patched to 2.6.3.2-1
When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is sending the job is resolving the {{hdp.version}} to its own version. The box that the job runs on, however, might not have that specific versioned directory.
The problem can only be solved externally by providing a way to ensure that the native libraries are on every machine which could potentially run jobs. We could install bits on every host, but that would defeat the space-saving purposes of patch upgrades.
Another solution is to ensure that the Tez and MapR tarballs have the required libraries and that those libraries are added to the classpath of launched jobs.
Diffs
-----
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml a7d8cd67bd
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 1427a6faae
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml 4ad08cebf9
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0df0334c53
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b4e3745099
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 98223904c8
Diff: https://reviews.apache.org/r/63684/diff/1/
Testing
-------
Manual upgrade on patched cluster.
Thanks,
Jonathan Hurley
Re: Review Request 63684: Patch Upgrades Broken For Clients Due To
Versioned LD Library
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190532
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Grinenko
On Nov. 8, 2017, 9:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2017, 9:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When services are patched to different versions, jobs being run in containers are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in via the job.
>
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
>
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is sending the job is resolving the {{hdp.version}} to its own version. The box that the job runs on, however, might not have that specific versioned directory.
>
> The problem can only be solved externally by providing a way to ensure that the native libraries are on every machine which could potentially run jobs. We could install bits on every host, but that would defeat the space-saving purposes of patch upgrades.
>
> Another solution is to ensure that the Tez and MapR tarballs have the required libraries and that those libraries are added to the classpath of launched jobs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml a7d8cd67bd
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 1427a6faae
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml 4ad08cebf9
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0df0334c53
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b4e3745099
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 98223904c8
>
>
> Diff: https://reviews.apache.org/r/63684/diff/1/
>
>
> Testing
> -------
>
> Manual upgrade on patched cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63684: Patch Upgrades Broken For Clients Due To
Versioned LD Library
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190553
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 8, 2017, 11:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2017, 11:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When services are patched to different versions, jobs being run in containers are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in via the job.
>
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
>
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is sending the job is resolving the {{hdp.version}} to its own version. The box that the job runs on, however, might not have that specific versioned directory.
>
> The problem can only be solved externally by providing a way to ensure that the native libraries are on every machine which could potentially run jobs. We could install bits on every host, but that would defeat the space-saving purposes of patch upgrades.
>
> Another solution is to ensure that the Tez and MapR tarballs have the required libraries and that those libraries are added to the classpath of launched jobs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml a7d8cd67bd
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 1427a6faae
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml 4ad08cebf9
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0df0334c53
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b4e3745099
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 98223904c8
>
>
> Diff: https://reviews.apache.org/r/63684/diff/2/
>
>
> Testing
> -------
>
> Manual upgrade on patched cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63684: Patch Upgrades Broken For Clients Due To
Versioned LD Library
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Nov. 9, 2017, 12:30 p.m., Nate Cole wrote:
> > Do we need to consider other upgrades, e.g. from HDP-2.5 to -2.6 as well?
It's not wrong to, but I don't think it's needed either. If you go from HDP 2.5 to 2.6, and then start patching on 2.6, it'll get changed at that point...
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190616
-----------------------------------------------------------
On Nov. 8, 2017, 4:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2017, 4:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When services are patched to different versions, jobs being run in containers are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in via the job.
>
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
>
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is sending the job is resolving the {{hdp.version}} to its own version. The box that the job runs on, however, might not have that specific versioned directory.
>
> The problem can only be solved externally by providing a way to ensure that the native libraries are on every machine which could potentially run jobs. We could install bits on every host, but that would defeat the space-saving purposes of patch upgrades.
>
> Another solution is to ensure that the Tez and MapR tarballs have the required libraries and that those libraries are added to the classpath of launched jobs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml a7d8cd67bd
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 1427a6faae
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml 4ad08cebf9
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0df0334c53
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b4e3745099
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 98223904c8
>
>
> Diff: https://reviews.apache.org/r/63684/diff/2/
>
>
> Testing
> -------
>
> Manual upgrade on patched cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63684: Patch Upgrades Broken For Clients Due To
Versioned LD Library
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63684/#review190616
-----------------------------------------------------------
Ship it!
Do we need to consider other upgrades, e.g. from HDP-2.5 to -2.6 as well?
- Nate Cole
On Nov. 8, 2017, 4:22 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63684/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2017, 4:22 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22386
> https://issues.apache.org/jira/browse/AMBARI-22386
>
>
> Repository: ambari
>
>
> Description
> -------
>
> When services are patched to different versions, jobs being run in containers are unable to pickup the correct JARs from the {{LD_LIBRARY_PATH}} passed in via the job.
>
> Consider the following situation:
> - Base cluster installed at 2.6.3.0-1
> - Enable some sort of codec, like snappy
> - YARN patched to 2.6.3.1-1
> - Hive patched to 2.6.3.2-1
>
> When Hive goes to run jobs on Tez or when jobs are run on MapR, whatever is sending the job is resolving the {{hdp.version}} to its own version. The box that the job runs on, however, might not have that specific versioned directory.
>
> The problem can only be solved externally by providing a way to ensure that the native libraries are on every machine which could potentially run jobs. We could install bits on every host, but that would defeat the space-saving purposes of patch upgrades.
>
> Another solution is to ensure that the Tez and MapR tarballs have the required libraries and that those libraries are added to the classpath of launched jobs.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/configuration-mapred/mapred-site.xml a7d8cd67bd
> ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml 1427a6faae
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml 4ad08cebf9
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0df0334c53
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml b4e3745099
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 98223904c8
>
>
> Diff: https://reviews.apache.org/r/63684/diff/2/
>
>
> Testing
> -------
>
> Manual upgrade on patched cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>