You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Mark Wagner <wa...@gmail.com> on 2013/11/02 02:17:08 UTC
Review Request 15194: Support multiple inputs for PigProcessor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------
Review request for pig, Cheolsoo Park and Daniel Dai.
Bugs: PIG-3527
https://issues.apache.org/jira/browse/PIG-3527
Repository: pig-git
Description
-------
Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
Diffs
-----
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
src/org/apache/pig/data/BinInterSedes.java b3ec51e
src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
test/e2e/pig/tests/tez.conf 24af8d3
Diff: https://reviews.apache.org/r/15194/diff/
Testing
-------
Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
Thanks,
Mark Wagner
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28185
-----------------------------------------------------------
- Daniel Dai
On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2013, 1:17 a.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3527
> https://issues.apache.org/jira/browse/PIG-3527
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
> src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
> src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
> src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
> src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
> src/org/apache/pig/data/BinInterSedes.java b3ec51e
> src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
> test/e2e/pig/tests/tez.conf 24af8d3
>
> Diff: https://reviews.apache.org/r/15194/diff/
>
>
> Testing
> -------
>
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28080
-----------------------------------------------------------
Looks good overall. I made a few minor comments below:
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54652>
Please remove white spaces in the patch.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54642>
Do we need this field? It doesn't seem used anywhere.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54643>
Private?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54644>
Private?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54645>
This constructor doesn't seem needed. Can we delete it?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54659>
Perhaps we add a copy constructor to POPackage and call it here? Just a thought.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54647>
Please remove unnecessary empty lines.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54646>
Can you run two for loops inside a single try-catch block instead of duplicating it?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54648>
If you keep the index of minimum instead of minimum itself, then you don't need to call comparator.compare() here, no?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54649>
Do we need this? Doesn't seem used anywhere.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54650>
Do we need this? Doesn't seem used anywhere.
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54651>
You can initialize inputKeys in declaration and remove these lines, can't you?
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54660>
Can't you do "tezOp.plan.replace(pack, shuffleLoad)" instead of remove and add? Then, you don't need to copy the succsList and re-connect them to shuffleLoad at all, no?
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54655>
Remove.
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54658>
If we use tezOp.plan.replace() below, we don't need to copy the ldSucs, no?
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54656>
Remove commented out code.
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54657>
Looks like this can be rewritten as follows:
POSimpleTezLoad tezLoad = new POSimpleTezLoad(new OperatorKey(scope , nig.getNextNodeId(scope)));
tezLoad.setInputKey(ld.getOperatorKey().toString());
tezOp.plan.replace(ld, tezLoad);
- Cheolsoo Park
On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2013, 1:17 a.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3527
> https://issues.apache.org/jira/browse/PIG-3527
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
> src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
> src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
> src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
> src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
> src/org/apache/pig/data/BinInterSedes.java b3ec51e
> src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
> test/e2e/pig/tests/tez.conf 24af8d3
>
> Diff: https://reviews.apache.org/r/15194/diff/
>
>
> Testing
> -------
>
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28179
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54825>
Need to take care of "pig.sortOrder"?
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54826>
We don't tag the input, so no index is needed. We might be able to further optimize this (but I am fine to leave it to the future).
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54824>
Now we no longer tag input, so no need to specify a group comparator (in old group comparator, we skip tag when compare, to make sure same key (even different tag) is grouped together.
- Daniel Dai
On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2013, 1:17 a.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3527
> https://issues.apache.org/jira/browse/PIG-3527
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
> src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
> src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
> src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
> src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
> src/org/apache/pig/data/BinInterSedes.java b3ec51e
> src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
> test/e2e/pig/tests/tez.conf 24af8d3
>
> Diff: https://reviews.apache.org/r/15194/diff/
>
>
> Testing
> -------
>
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Mark Wagner <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------
(Updated Nov. 28, 2013, 12:41 a.m.)
Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
Changes
-------
Update with POPackage refactoring (see PIG-3595). There are still some items from previous reviews that I have not completed.
Bugs: PIG-3527
https://issues.apache.org/jira/browse/PIG-3527
Repository: pig-git
Description
-------
Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
Diffs (updated)
-----
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java 3638b5c
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java fc9261c
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 9341153
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 93de6d5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 9580c26
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java 64f0ee1
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java 933363d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java 906930a
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java eea5ce3
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java 1578630
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java 47137d5
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java abb16ff
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java ff82801
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java 892c26f
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java 9105a0e
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java 82f11ac
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java d604174
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java b860521
src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 68610dd
src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 87b198f
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 3dcefa9
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/TezPOPackageAnnotator.java a57ac45
src/org/apache/pig/backend/hadoop/executionengine/util/CombinerOptimizerUtil.java b163184
src/org/apache/pig/data/BinInterSedes.java b3ec51e
src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
src/org/apache/pig/data/ReadOnceBag.java e2b3887
src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 7112695
src/org/apache/pig/pen/IllustratorAttacher.java db9c707
src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7
test/e2e/pig/tests/tez.conf 4928a8e
test/org/apache/pig/test/TestJobSubmission.java cccade4
test/org/apache/pig/test/TestPackage.java 6b197d1
test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a
test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728
test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f
test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d
test/org/apache/pig/test/data/GoldenFiles/MRC12.gld a5c5a1b
test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b
test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509
test/org/apache/pig/test/data/GoldenFiles/MRC15.gld cce2523
test/org/apache/pig/test/data/GoldenFiles/MRC16.gld c319c87
test/org/apache/pig/test/data/GoldenFiles/MRC17.gld 7e92e31
test/org/apache/pig/test/data/GoldenFiles/MRC18.gld d13e3ab
test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 569fd21
test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0
test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f
test/org/apache/pig/test/data/GoldenFiles/MRC4.gld e0bf184
test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 25dc2c3
test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 1b8e9c0
test/org/apache/pig/test/data/GoldenFiles/MRC8.gld f3beba8
test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c162b2d
Diff: https://reviews.apache.org/r/15194/diff/
Testing
-------
Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
Thanks,
Mark Wagner
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28079
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54640>
This breaks compilation with Hadoop 1.x.
i.e. -Dhadoopversion=20
We need to move it under tez package.
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
<https://reviews.apache.org/r/15194/#comment54641>
This reverts PIG-3060 and breaks TestEvalPipelineLocal.testFlattenEmptyBag.
https://issues.apache.org/jira/browse/PIG-3060
- Cheolsoo Park
On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2013, 1:17 a.m.)
>
>
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3527
> https://issues.apache.org/jira/browse/PIG-3527
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
> src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
> src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
> src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
> src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
> src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
> src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
> src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
> src/org/apache/pig/data/BinInterSedes.java b3ec51e
> src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
> test/e2e/pig/tests/tez.conf 24af8d3
>
> Diff: https://reviews.apache.org/r/15194/diff/
>
>
> Testing
> -------
>
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
>
>
> Thanks,
>
> Mark Wagner
>
>
Re: Review Request 15194: Support multiple inputs for PigProcessor
Posted by Mark Wagner <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------
(Updated Nov. 2, 2013, 1:17 a.m.)
Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
Bugs: PIG-3527
https://issues.apache.org/jira/browse/PIG-3527
Repository: pig-git
Description
-------
Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
Diffs
-----
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715
src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330
src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145
src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0
src/org/apache/pig/data/BinInterSedes.java b3ec51e
src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
test/e2e/pig/tests/tez.conf 24af8d3
Diff: https://reviews.apache.org/r/15194/diff/
Testing
-------
Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
Thanks,
Mark Wagner