You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Adam Szita via Review Board <no...@reviews.apache.org> on 2020/04/06 14:57:57 UTC
Review Request 72326: PIG-5253: Pig Hadoop 3 support
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72326/
-----------------------------------------------------------
Review request for pig, Koji Noguchi, Nandor Kollar, and Rohini Palaniswamy.
Bugs: PIG-5253
https://issues.apache.org/jira/browse/PIG-5253
Repository: pig
Description
-------
This it continuing from https://reviews.apache.org/r/65239/ (there's issues with reviewboard's pig-git repo)
This change is now rebased on trunk, and I fixed test issues around the MR mode MiniGenericCluster refactoring.
Diffs
-----
trunk/bin/pig 1876187
trunk/bin/pig.py 1876187
trunk/build.xml 1876187
trunk/ivy.xml 1876187
trunk/ivy/libraries-h2.properties PRE-CREATION
trunk/ivy/libraries-h3.properties PRE-CREATION
trunk/ivy/libraries.properties 1876187
trunk/test/org/apache/pig/parser/TestErrorHandling.java 1876187
trunk/test/org/apache/pig/parser/TestQueryParserUtils.java 1876187
trunk/test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION
trunk/test/org/apache/pig/test/MiniCluster.java 1876187
trunk/test/org/apache/pig/test/MiniGenericCluster.java 1876187
trunk/test/org/apache/pig/test/SparkMiniCluster.java 1876187
trunk/test/org/apache/pig/test/TestGrunt.java 1876187
trunk/test/org/apache/pig/test/TezMiniCluster.java 1876187
trunk/test/org/apache/pig/test/Util.java 1876187
trunk/test/org/apache/pig/test/YarnMiniCluster.java 1876187
Diff: https://reviews.apache.org/r/72326/diff/1/
Testing
-------
Thanks,
Adam Szita
Re: Review Request 72326: PIG-5253: Pig Hadoop 3 support
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72326/#review224613
-----------------------------------------------------------
trunk/test/org/apache/pig/test/MapReduceMiniCluster.java
Lines 1 (patched)
<https://reviews.apache.org/r/72326/#comment313397>
Koji found that this file has the Cloudera license. Can you replace with Apache license?
trunk/test/org/apache/pig/test/MapReduceMiniCluster.java
Lines 41 (patched)
<https://reviews.apache.org/r/72326/#comment313399>
m_conf.set("dfs.datanode.address", "0.0.0.0:0");
m_conf.set("dfs.datanode.http.address", "0.0.0.0:0");
m_conf.set("pig.jobcontrol.sleep", "100");
System.setProperty("cluster", m_conf.get(MRConfiguration.JOB_TRACKER));
System.setProperty("namenode", m_conf.get(FileSystem.FS_DEFAULT_NAME_KEY));
is missing compared to older MiniCluster.java. Not sure datanode address settings are needed but setting pig.jobcontrol.sleep is likely needed to have the tests run faster.
trunk/test/org/apache/pig/test/TezMiniCluster.java
Line 61 (original), 65 (patched)
<https://reviews.apache.org/r/72326/#comment313398>
Can you add
tez_conf.set("tez.runtime.transfer.data-via-events.enabled", "false");
here. Koji found that some tests were failing with Hadoop 3 in local mode without that setting.
- Rohini Palaniswamy
On April 6, 2020, 2:57 p.m., Adam Szita wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72326/
> -----------------------------------------------------------
>
> (Updated April 6, 2020, 2:57 p.m.)
>
>
> Review request for pig, Koji Noguchi, Nandor Kollar, and Rohini Palaniswamy.
>
>
> Bugs: PIG-5253
> https://issues.apache.org/jira/browse/PIG-5253
>
>
> Repository: pig
>
>
> Description
> -------
>
> This it continuing from https://reviews.apache.org/r/65239/ (there's issues with reviewboard's pig-git repo)
> This change is now rebased on trunk, and I fixed test issues around the MR mode MiniGenericCluster refactoring.
>
>
> Diffs
> -----
>
> trunk/bin/pig 1876187
> trunk/bin/pig.py 1876187
> trunk/build.xml 1876187
> trunk/ivy.xml 1876187
> trunk/ivy/libraries-h2.properties PRE-CREATION
> trunk/ivy/libraries-h3.properties PRE-CREATION
> trunk/ivy/libraries.properties 1876187
> trunk/test/org/apache/pig/parser/TestErrorHandling.java 1876187
> trunk/test/org/apache/pig/parser/TestQueryParserUtils.java 1876187
> trunk/test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION
> trunk/test/org/apache/pig/test/MiniCluster.java 1876187
> trunk/test/org/apache/pig/test/MiniGenericCluster.java 1876187
> trunk/test/org/apache/pig/test/SparkMiniCluster.java 1876187
> trunk/test/org/apache/pig/test/TestGrunt.java 1876187
> trunk/test/org/apache/pig/test/TezMiniCluster.java 1876187
> trunk/test/org/apache/pig/test/Util.java 1876187
> trunk/test/org/apache/pig/test/YarnMiniCluster.java 1876187
>
>
> Diff: https://reviews.apache.org/r/72326/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Adam Szita
>
>