You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Nandor Kollar via Review Board <no...@reviews.apache.org> on 2019/04/10 11:49:46 UTC
Re: Review Request 65239: PIG-5253: Pig Hadoop 3 support
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65239/
-----------------------------------------------------------
(Updated April 10, 2019, 11:49 a.m.)
Review request for pig, Daniel Dai, Koji Noguchi, Rohini Palaniswamy, and Adam Szita.
Changes
-------
- Hadoop 2 jar is created even when build is hadoop 3
- missing important h3 dependencies for local mode
Repository: pig-git
Description
-------
This is an initial patch that adds Hadoop 3 support to Pig in addition to Hadoop 2.
Major modifications:
* No breaking API change was introduced in Hadoop 3, the current code compiles with Hadoop 3
* hadoopversion property tells which mode the tests should run, the default is hadoop 2
* Hadoop 3 introduced a security fix, only whitelisted environment variables are passed to MiniCluster
* In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml, hdfs-site.xml and mapred-site.xml. I decided to write the config into all of these files in MiniCluster.java (into hadoop-site.xml too to stay compatible with Hadoop 2) for the shake of simplicity, we might want to have different files for Hadoop 2 and separate the properties for Hadoop 3.
* TestErrorHandling.java: small format change in error message, modified the assert so it works on both on Hadoop 2 and Hadoop 3
* HadoopShims: code is identical with Hadoop 2, not sure if we need shims any more. I think we should move it to the src instead.
* Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3 specific properties
* ivy.xml: new config for Hadoop 3
* build.xml: new target to package both hadoop2 and hadoop3 - not sure that this is needed, if we move shims, the I think we don't need this target
* HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html HBase 1.x is not tested against Hadoop 2)
Diffs (updated)
-----
bin/pig 3fcf165106cccbe75fc1c61ea74732456ae50fc7
bin/pig.py b6c396579c54359f430c6e74d055ec7f27ae2197
build.xml 9bb1b125eef3c9381466b8f043ecd31ff0d56dc7
ivy.xml a93da2a9a20edf103dcbf08abc92301a1c2fda9e
ivy/libraries-h2.properties PRE-CREATION
ivy/libraries-h3.properties PRE-CREATION
ivy/libraries.properties 1abc9677e6c0747bb102c563b4f6642274541476
test/org/apache/pig/parser/TestErrorHandling.java 15e09031c360cea5f81609129ac3a6d38d68d3ea
test/org/apache/pig/parser/TestQueryParserUtils.java 1c217e3cab9c4b5dc51289a883aa696dcd2feeea
test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION
test/org/apache/pig/test/MiniCluster.java a7532ad750f06ffae5a03024b1658ff77152c902
test/org/apache/pig/test/MiniGenericCluster.java 674860f880407595d68c4eea2b67e2d6465417fe
test/org/apache/pig/test/Util.java 788a72fe3ceca08ec61ae425a393b5b0936454f4
test/org/apache/pig/test/YarnMiniCluster.java 69d808124a4e9be661f1fda25755075dcb6607b1
Diff: https://reviews.apache.org/r/65239/diff/4/
Changes: https://reviews.apache.org/r/65239/diff/3-4/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 65239: PIG-5253: Pig Hadoop 3 support
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65239/#review221505
-----------------------------------------------------------
Ship it!
It would be good to update to the latest hadoop versions - 2.10.0 and 3.3.0. But have let this one languish for long without review. So lets get this in and update to new versions in a separate jira as 3.3.0 may require some additional changes.
- Rohini Palaniswamy
On April 10, 2019, 11:49 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65239/
> -----------------------------------------------------------
>
> (Updated April 10, 2019, 11:49 a.m.)
>
>
> Review request for pig, Daniel Dai, Koji Noguchi, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> This is an initial patch that adds Hadoop 3 support to Pig in addition to Hadoop 2.
>
> Major modifications:
> * No breaking API change was introduced in Hadoop 3, the current code compiles with Hadoop 3
> * hadoopversion property tells which mode the tests should run, the default is hadoop 2
> * Hadoop 3 introduced a security fix, only whitelisted environment variables are passed to MiniCluster
> * In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml, hdfs-site.xml and mapred-site.xml. I decided to write the config into all of these files in MiniCluster.java (into hadoop-site.xml too to stay compatible with Hadoop 2) for the shake of simplicity, we might want to have different files for Hadoop 2 and separate the properties for Hadoop 3.
> * TestErrorHandling.java: small format change in error message, modified the assert so it works on both on Hadoop 2 and Hadoop 3
> * HadoopShims: code is identical with Hadoop 2, not sure if we need shims any more. I think we should move it to the src instead.
> * Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3 specific properties
> * ivy.xml: new config for Hadoop 3
> * build.xml: new target to package both hadoop2 and hadoop3 - not sure that this is needed, if we move shims, the I think we don't need this target
> * HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html HBase 1.x is not tested against Hadoop 2)
>
>
> Diffs
> -----
>
> bin/pig 3fcf165106cccbe75fc1c61ea74732456ae50fc7
> bin/pig.py b6c396579c54359f430c6e74d055ec7f27ae2197
> build.xml 9bb1b125eef3c9381466b8f043ecd31ff0d56dc7
> ivy.xml a93da2a9a20edf103dcbf08abc92301a1c2fda9e
> ivy/libraries-h2.properties PRE-CREATION
> ivy/libraries-h3.properties PRE-CREATION
> ivy/libraries.properties 1abc9677e6c0747bb102c563b4f6642274541476
> test/org/apache/pig/parser/TestErrorHandling.java 15e09031c360cea5f81609129ac3a6d38d68d3ea
> test/org/apache/pig/parser/TestQueryParserUtils.java 1c217e3cab9c4b5dc51289a883aa696dcd2feeea
> test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION
> test/org/apache/pig/test/MiniCluster.java a7532ad750f06ffae5a03024b1658ff77152c902
> test/org/apache/pig/test/MiniGenericCluster.java 674860f880407595d68c4eea2b67e2d6465417fe
> test/org/apache/pig/test/Util.java 788a72fe3ceca08ec61ae425a393b5b0936454f4
> test/org/apache/pig/test/YarnMiniCluster.java 69d808124a4e9be661f1fda25755075dcb6607b1
>
>
> Diff: https://reviews.apache.org/r/65239/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>