You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/11/14 21:21:34 UTC
Review Request 63807: Add Native Libraries To Tez Tarball
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22444
https://issues.apache.org/jira/browse/AMBARI-22444
Repository: ambari
Description
-------
As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
{code}
In order to fix this, the Tez tarball should include the native hadoop libraries as well:
{code}
??? tez
? ??? lib
? ? ??? native
? ? ? ??? libhadoop.a
? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
? ? ? ??? libhadoop.so.1.0.0
? ? ? ??? libhadooppipes.a
? ? ? ??? libhadooputils.a
? ? ? ??? libhdfs.a
? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
? ? ? ??? libhdfs.so.0.0.0
? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
? ? ? ??? libsnappy.so.1.1.4
{code}
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
Diff: https://reviews.apache.org/r/63807/diff/1/
Testing
-------
Manual testing to ensure the file is created and uploaded.
UNIT TESTS PENDING...
Thanks,
Jonathan Hurley
Re: Review Request 63807: Add Native Libraries To Tez Tarball
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
-----------------------------------------------------------
Fix it, then Ship it!
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)
<https://reviews.apache.org/r/63807/#comment268602>
this is a bit dangerous, as would return the same function and calling it would make infinite recursion.
better replace to something like that, by skipping values which r not needed:
_, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
_, tez_source_file, _, _ = get_tarball_paths("tez")
Round brackets do not needed here
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)
<https://reviews.apache.org/r/63807/#comment268596>
why we trying to copy directory, if it could be even not exists?
SHoudk we fail at line 70 then?
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)
<https://reviews.apache.org/r/63807/#comment268613>
I know, they were here, but that is not pythonic. Please remove round brackets, they r reducant
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)
<https://reviews.apache.org/r/63807/#comment268614>
I know, they were here, but that is not pythonic. Please remove round brackets, they r reducant
- Dmytro Grinenko
On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2017, 9:21 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
>
>
> Repository: ambari
>
>
> Description
> -------
>
> As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
>
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
>
> When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> {code}
>
> In order to fix this, the Tez tarball should include the native hadoop libraries as well:
> {code}
> ??? tez
> ? ??? lib
> ? ? ??? native
> ? ? ? ??? libhadoop.a
> ? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
> ? ? ? ??? libhadoop.so.1.0.0
> ? ? ? ??? libhadooppipes.a
> ? ? ? ??? libhadooputils.a
> ? ? ? ??? libhdfs.a
> ? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
> ? ? ? ??? libhdfs.so.0.0.0
> ? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1.1.4
> {code}
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
>
>
> Diff: https://reviews.apache.org/r/63807/diff/1/
>
>
> Testing
> -------
>
> Manual testing to ensure the file is created and uploaded.
>
> UNIT TESTS PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63807: Add Native Libraries To Tez Tarball
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191046
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Nov. 14, 2017, 11:21 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2017, 11:21 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
>
>
> Repository: ambari
>
>
> Description
> -------
>
> As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
>
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
>
> When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> {code}
>
> In order to fix this, the Tez tarball should include the native hadoop libraries as well:
> {code}
> ??? tez
> ? ??? lib
> ? ? ??? native
> ? ? ? ??? libhadoop.a
> ? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
> ? ? ? ??? libhadoop.so.1.0.0
> ? ? ? ??? libhadooppipes.a
> ? ? ? ??? libhadooputils.a
> ? ? ? ??? libhdfs.a
> ? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
> ? ? ? ??? libhdfs.so.0.0.0
> ? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1.1.4
> {code}
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
>
>
> Diff: https://reviews.apache.org/r/63807/diff/1/
>
>
> Testing
> -------
>
> Manual testing to ensure the file is created and uploaded.
>
> UNIT TESTS PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63807: Add Native Libraries To Tez Tarball
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/
-----------------------------------------------------------
(Updated Nov. 15, 2017, 9:07 a.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22444
https://issues.apache.org/jira/browse/AMBARI-22444
Repository: ambari
Description
-------
As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
{code}
In order to fix this, the Tez tarball should include the native hadoop libraries as well:
{code}
??? tez
? ??? lib
? ? ??? native
? ? ? ??? libhadoop.a
? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
? ? ? ??? libhadoop.so.1.0.0
? ? ? ??? libhadooppipes.a
? ? ? ??? libhadooputils.a
? ? ? ??? libhdfs.a
? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
? ? ? ??? libhdfs.so.0.0.0
? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
? ? ? ??? libsnappy.so.1.1.4
{code}
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
Diff: https://reviews.apache.org/r/63807/diff/2/
Changes: https://reviews.apache.org/r/63807/diff/1-2/
Testing
-------
Manual testing to ensure the file is created and uploaded.
UNIT TESTS PENDING...
Thanks,
Jonathan Hurley
Re: Review Request 63807: Add Native Libraries To Tez Tarball
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191022
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Nov. 14, 2017, 4:21 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2017, 4:21 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
>
>
> Repository: ambari
>
>
> Description
> -------
>
> As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
>
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
>
> When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> {code}
>
> In order to fix this, the Tez tarball should include the native hadoop libraries as well:
> {code}
> ??? tez
> ? ??? lib
> ? ? ??? native
> ? ? ? ??? libhadoop.a
> ? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
> ? ? ? ??? libhadoop.so.1.0.0
> ? ? ? ??? libhadooppipes.a
> ? ? ? ??? libhadooputils.a
> ? ? ? ??? libhdfs.a
> ? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
> ? ? ? ??? libhdfs.so.0.0.0
> ? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1.1.4
> {code}
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
>
>
> Diff: https://reviews.apache.org/r/63807/diff/1/
>
>
> Testing
> -------
>
> Manual testing to ensure the file is created and uploaded.
>
> UNIT TESTS PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 63807: Add Native Libraries To Tez Tarball
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191012
-----------------------------------------------------------
- Dmytro Grinenko
On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2017, 9:21 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
>
>
> Repository: ambari
>
>
> Description
> -------
>
> As part of the investigation for getting patch upgrades working in Ambari 2.6.1, it was determined that the Tez tarball will need to have the hadoop native libraries added to it so that they can be detected from the tarball.
>
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
>
> When Hive commands run, they will attempt to load the native snappy libraries from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
> at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217)
> {code}
>
> In order to fix this, the Tez tarball should include the native hadoop libraries as well:
> {code}
> ??? tez
> ? ??? lib
> ? ? ??? native
> ? ? ? ??? libhadoop.a
> ? ? ? ??? libhadoop.so -> libhadoop.so.1.0.0
> ? ? ? ??? libhadoop.so.1.0.0
> ? ? ? ??? libhadooppipes.a
> ? ? ? ??? libhadooputils.a
> ? ? ? ??? libhdfs.a
> ? ? ? ??? libhdfs.so -> libhdfs.so.0.0.0
> ? ? ? ??? libhdfs.so.0.0.0
> ? ? ? ??? libsnappy.so -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ? ? ? ??? libsnappy.so.1.1.4
> {code}
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py 03b6213ac4
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py c682c3e24a
>
>
> Diff: https://reviews.apache.org/r/63807/diff/1/
>
>
> Testing
> -------
>
> Manual testing to ensure the file is created and uploaded.
>
> UNIT TESTS PENDING...
>
>
> Thanks,
>
> Jonathan Hurley
>
>