You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2014/02/18 09:28:52 UTC
Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting and
closing of inputs and outputs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/
-----------------------------------------------------------
Review request for pig, Cheolsoo Park and Daniel Dai.
Bugs: PIG-3767
https://issues.apache.org/jira/browse/PIG-3767
Repository: pig
Description
-------
Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
Diffs
-----
http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
Diff: https://reviews.apache.org/r/18203/diff/
Testing
-------
tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
Thanks,
Rohini Palaniswamy
Re: Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting
and closing of inputs and outputs
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/#review34683
-----------------------------------------------------------
Ship it!
Otherwise +1, feel free to commit.
http://svn.apache.org/repos/asf/pig/branches/tez/build.xml
<https://reviews.apache.org/r/18203/#comment64905>
The condition you check is hadoopversion=23, should name it guava-hadoop2.version?
- Daniel Dai
On Feb. 18, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18203/
> -----------------------------------------------------------
>
> (Updated Feb. 18, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3767
> https://issues.apache.org/jira/browse/PIG-3767
>
>
> Repository: pig
>
>
> Description
> -------
>
> Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
>
> Diff: https://reviews.apache.org/r/18203/diff/
>
>
> Testing
> -------
>
> tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>
Re: Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting
and closing of inputs and outputs
Posted by Cheolsoo Park <pi...@gmail.com>.
> On Feb. 18, 2014, 12:21 p.m., Cheolsoo Park wrote:
> > Can we hold on please?
> >
> > If we commit this, can I still compile and run Pig with Tez that is prior to TEZ-688? Due to the Guava conflict (TEZ-787), I cannot upgrade Tez to the head. So I'd like to avoid any incompatible changes before TEZ-787 is resolved.
> >
I meant TEZ-646 not TEZ-688 that bumped Guava to 0.15.
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/#review34688
-----------------------------------------------------------
On Feb. 18, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18203/
> -----------------------------------------------------------
>
> (Updated Feb. 18, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3767
> https://issues.apache.org/jira/browse/PIG-3767
>
>
> Repository: pig
>
>
> Description
> -------
>
> Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
>
> Diff: https://reviews.apache.org/r/18203/diff/
>
>
> Testing
> -------
>
> tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>
Re: Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting
and closing of inputs and outputs
Posted by Cheolsoo Park <pi...@gmail.com>.
> On Feb. 18, 2014, 12:21 p.m., Cheolsoo Park wrote:
> > Can we hold on please?
> >
> > If we commit this, can I still compile and run Pig with Tez that is prior to TEZ-688? Due to the Guava conflict (TEZ-787), I cannot upgrade Tez to the head. So I'd like to avoid any incompatible changes before TEZ-787 is resolved.
> >
>
> Cheolsoo Park wrote:
> I meant TEZ-646 not TEZ-688 that bumped Guava to 0.15.
>
> Rohini Palaniswamy wrote:
> I missed this comment and just committed. Can revert back. But with this change anyways, you cannot compile and run pig with Tez that is prior to TEZ-688 as it also calls input.start() and output.start() which are new APIs in tez trunk.
Talked to Rohini offline. I will temporarily rebase my Pig repo until Tez resolves TEZ-787.
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/#review34688
-----------------------------------------------------------
On Feb. 18, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18203/
> -----------------------------------------------------------
>
> (Updated Feb. 18, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3767
> https://issues.apache.org/jira/browse/PIG-3767
>
>
> Repository: pig
>
>
> Description
> -------
>
> Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
>
> Diff: https://reviews.apache.org/r/18203/diff/
>
>
> Testing
> -------
>
> tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>
Re: Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting
and closing of inputs and outputs
Posted by Rohini Palaniswamy <ro...@gmail.com>.
> On Feb. 18, 2014, 12:21 p.m., Cheolsoo Park wrote:
> > Can we hold on please?
> >
> > If we commit this, can I still compile and run Pig with Tez that is prior to TEZ-688? Due to the Guava conflict (TEZ-787), I cannot upgrade Tez to the head. So I'd like to avoid any incompatible changes before TEZ-787 is resolved.
> >
>
> Cheolsoo Park wrote:
> I meant TEZ-646 not TEZ-688 that bumped Guava to 0.15.
I missed this comment and just committed. Can revert back. But with this change anyways, you cannot compile and run pig with Tez that is prior to TEZ-688 as it also calls input.start() and output.start() which are new APIs in tez trunk.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/#review34688
-----------------------------------------------------------
On Feb. 18, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18203/
> -----------------------------------------------------------
>
> (Updated Feb. 18, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3767
> https://issues.apache.org/jira/browse/PIG-3767
>
>
> Repository: pig
>
>
> Description
> -------
>
> Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
>
> Diff: https://reviews.apache.org/r/18203/diff/
>
>
> Testing
> -------
>
> tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>
Re: Review Request 18203: [PIG-3767] Work with TEZ-668 which allows starting
and closing of inputs and outputs
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18203/#review34688
-----------------------------------------------------------
Can we hold on please?
If we commit this, can I still compile and run Pig with Tez that is prior to TEZ-688? Due to the Guava conflict (TEZ-787), I cannot upgrade Tez to the head. So I'd like to avoid any incompatible changes before TEZ-787 is resolved.
- Cheolsoo Park
On Feb. 18, 2014, 8:28 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18203/
> -----------------------------------------------------------
>
> (Updated Feb. 18, 2014, 8:28 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3767
> https://issues.apache.org/jira/browse/PIG-3767
>
>
> Repository: pig
>
>
> Description
> -------
>
> Work with latest Tez trunk and incorporate tez breaking changes for Input and Output.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/build.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy.xml 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/ivy/libraries.properties 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POSimpleTezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1568909
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/builtin/ReadScalarsTez.java 1568909
>
> Diff: https://reviews.apache.org/r/18203/diff/
>
>
> Testing
> -------
>
> tez.conf e2e tests pass and run fast. test-tez unit tests take a really long time to run. Need to investigate and fix. Will do that in a separate jira.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>