You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2014/02/07 02:03:30 UTC
Review Request 17824: [PIG-3748] Support for multiquery off in Tez
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17824/
-----------------------------------------------------------
Review request for pig, Cheolsoo Park and Daniel Dai.
Bugs: PIG-3748
https://issues.apache.org/jira/browse/PIG-3748
Repository: pig
Description
-------
When multiquery is off, POSplit is removed from the vertex and the tuple generated before POSplit is written once to each output and the plan after POSplit is executed in the subsequent vertex instead of being executed as a sub-plan of POSplit in the same vertex(multiquery on scenario).
Optimizations TODO:
1) Once Tez supports shared edge, write once in POValueOutTez and let it be available for all the downstream vertices consuming the data.
2) POValueOutTez write key,value now with the key being empty. Weite a Input Output in Tez which only supports values and avoid writing empty keys.
Diffs
-----
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/Main.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigServer.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueOutputTez.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestBatchAliases.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestGrunt.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestLoad.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQuery.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryBasic.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigRunner.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigServer.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestStore.java 1565502
http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1565502
Diff: https://reviews.apache.org/r/17824/diff/
Testing
-------
Manually tested. Will fix older unit tests or add new ones later.
Also Updated TezC7.gld file which was failing because of some earlier jira changes.
Thanks,
Rohini Palaniswamy
Re: Review Request 17824: [PIG-3748] Support for multiquery off in Tez
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17824/#review33884
-----------------------------------------------------------
Ship it!
Great you also make POValueInputTez and POValueOuputTez. Let's commit it so I can base my work on it.
- Daniel Dai
On Feb. 7, 2014, 1:03 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17824/
> -----------------------------------------------------------
>
> (Updated Feb. 7, 2014, 1:03 a.m.)
>
>
> Review request for pig, Cheolsoo Park and Daniel Dai.
>
>
> Bugs: PIG-3748
> https://issues.apache.org/jira/browse/PIG-3748
>
>
> Repository: pig
>
>
> Description
> -------
>
> When multiquery is off, POSplit is removed from the vertex and the tuple generated before POSplit is written once to each output and the plan after POSplit is executed in the subsequent vertex instead of being executed as a sub-plan of POSplit in the same vertex(multiquery on scenario).
>
> Optimizations TODO:
> 1) Once Tez supports shared edge, write once in POValueOutTez and let it be available for all the downstream vertices consuming the data.
> 2) POValueOutTez write key,value now with the key being empty. Weite a Input Output in Tez which only supports values and avoid writing empty keys.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/Main.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigServer.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueInputTez.java PRE-CREATION
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POValueOutputTez.java PRE-CREATION
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestBatchAliases.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestGrunt.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestLoad.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQuery.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryBasic.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryCompiler.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestMultiQueryLocal.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigRunner.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigServer.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestStore.java 1565502
> http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1565502
>
> Diff: https://reviews.apache.org/r/17824/diff/
>
>
> Testing
> -------
>
> Manually tested. Will fix older unit tests or add new ones later.
>
> Also Updated TezC7.gld file which was failing because of some earlier jira changes.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>