You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Mark Wagner <wa...@gmail.com> on 2013/11/02 02:17:08 UTC

Review Request 15194: Support multiple inputs for PigProcessor

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------

Review request for pig, Cheolsoo Park and Daniel Dai.


Bugs: PIG-3527
    https://issues.apache.org/jira/browse/PIG-3527


Repository: pig-git


Description
-------

Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).


Diffs
-----

  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
  src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
  src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
  src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
  src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
  src/org/apache/pig/data/BinInterSedes.java b3ec51e 
  src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
  test/e2e/pig/tests/tez.conf 24af8d3 

Diff: https://reviews.apache.org/r/15194/diff/


Testing
-------

Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.


Thanks,

Mark Wagner


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28185
-----------------------------------------------------------


- Daniel Dai


On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
> 
> (Updated Nov. 2, 2013, 1:17 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3527
>     https://issues.apache.org/jira/browse/PIG-3527
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
>   src/org/apache/pig/data/BinInterSedes.java b3ec51e 
>   src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
>   test/e2e/pig/tests/tez.conf 24af8d3 
> 
> Diff: https://reviews.apache.org/r/15194/diff/
> 
> 
> Testing
> -------
> 
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
> 
> 
> Thanks,
> 
> Mark Wagner
> 
>


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28080
-----------------------------------------------------------


Looks good overall. I made a few minor comments below:


src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54652>

    Please remove white spaces in the patch.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54642>

    Do we need this field? It doesn't seem used anywhere.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54643>

    Private?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54644>

    Private?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54645>

    This constructor doesn't seem needed. Can we delete it?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54659>

    Perhaps we add a copy constructor to POPackage and call it here? Just a thought.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54647>

    Please remove unnecessary empty lines.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54646>

    Can you run two for loops inside a single try-catch block instead of duplicating it?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54648>

    If you keep the index of minimum instead of minimum itself, then you don't need to call comparator.compare() here, no?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54649>

    Do we need this? Doesn't seem used anywhere.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54650>

    Do we need this? Doesn't seem used anywhere.



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54651>

    You can initialize inputKeys in declaration and remove these lines, can't you?



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54660>

    Can't you do "tezOp.plan.replace(pack, shuffleLoad)" instead of remove and add? Then, you don't need to copy the succsList and re-connect them to shuffleLoad at all, no?



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54655>

    Remove.



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54658>

    If we use tezOp.plan.replace() below, we don't need to copy the ldSucs, no?



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54656>

    Remove commented out code.



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54657>

    Looks like this can be rewritten as follows:
    
    POSimpleTezLoad tezLoad = new POSimpleTezLoad(new OperatorKey(scope , nig.getNextNodeId(scope)));
    tezLoad.setInputKey(ld.getOperatorKey().toString());
    tezOp.plan.replace(ld, tezLoad);


- Cheolsoo Park


On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
> 
> (Updated Nov. 2, 2013, 1:17 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3527
>     https://issues.apache.org/jira/browse/PIG-3527
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
>   src/org/apache/pig/data/BinInterSedes.java b3ec51e 
>   src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
>   test/e2e/pig/tests/tez.conf 24af8d3 
> 
> Diff: https://reviews.apache.org/r/15194/diff/
> 
> 
> Testing
> -------
> 
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
> 
> 
> Thanks,
> 
> Mark Wagner
> 
>


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28179
-----------------------------------------------------------



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54825>

    Need to take care of "pig.sortOrder"?



src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54826>

    We don't tag the input, so no index is needed. We might be able to further optimize this (but I am fine to leave it to the future).



src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
<https://reviews.apache.org/r/15194/#comment54824>

    Now we no longer tag input, so no need to specify a group comparator (in old group comparator, we skip tag when compare, to make sure same key (even different tag) is grouped together.


- Daniel Dai


On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
> 
> (Updated Nov. 2, 2013, 1:17 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3527
>     https://issues.apache.org/jira/browse/PIG-3527
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
>   src/org/apache/pig/data/BinInterSedes.java b3ec51e 
>   src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
>   test/e2e/pig/tests/tez.conf 24af8d3 
> 
> Diff: https://reviews.apache.org/r/15194/diff/
> 
> 
> Testing
> -------
> 
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
> 
> 
> Thanks,
> 
> Mark Wagner
> 
>


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Mark Wagner <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------

(Updated Nov. 28, 2013, 12:41 a.m.)


Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.


Changes
-------

Update with POPackage refactoring (see PIG-3595). There are still some items from previous reviews that I have not completed.


Bugs: PIG-3527
    https://issues.apache.org/jira/browse/PIG-3527


Repository: pig-git


Description
-------

Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).


Diffs (updated)
-----

  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java 3638b5c 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java fc9261c 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java 9341153 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 93de6d5 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 9580c26 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java 64f0ee1 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java 933363d 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java 906930a 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java eea5ce3 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java 1578630 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java 47137d5 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java abb16ff 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java ff82801 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java 892c26f 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java 9105a0e 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java 82f11ac 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java d604174 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java b860521 
  src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
  src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
  src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 68610dd 
  src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 87b198f 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 3dcefa9 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezPOPackageAnnotator.java a57ac45 
  src/org/apache/pig/backend/hadoop/executionengine/util/CombinerOptimizerUtil.java b163184 
  src/org/apache/pig/data/BinInterSedes.java b3ec51e 
  src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
  src/org/apache/pig/data/ReadOnceBag.java e2b3887 
  src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 7112695 
  src/org/apache/pig/pen/IllustratorAttacher.java db9c707 
  src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7 
  test/e2e/pig/tests/tez.conf 4928a8e 
  test/org/apache/pig/test/TestJobSubmission.java cccade4 
  test/org/apache/pig/test/TestPackage.java 6b197d1 
  test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a 
  test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728 
  test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f 
  test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d 
  test/org/apache/pig/test/data/GoldenFiles/MRC12.gld a5c5a1b 
  test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b 
  test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509 
  test/org/apache/pig/test/data/GoldenFiles/MRC15.gld cce2523 
  test/org/apache/pig/test/data/GoldenFiles/MRC16.gld c319c87 
  test/org/apache/pig/test/data/GoldenFiles/MRC17.gld 7e92e31 
  test/org/apache/pig/test/data/GoldenFiles/MRC18.gld d13e3ab 
  test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 569fd21 
  test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0 
  test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f 
  test/org/apache/pig/test/data/GoldenFiles/MRC4.gld e0bf184 
  test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 25dc2c3 
  test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 1b8e9c0 
  test/org/apache/pig/test/data/GoldenFiles/MRC8.gld f3beba8 
  test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c162b2d 

Diff: https://reviews.apache.org/r/15194/diff/


Testing
-------

Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.


Thanks,

Mark Wagner


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/#review28079
-----------------------------------------------------------



src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java
<https://reviews.apache.org/r/15194/#comment54640>

    This breaks compilation with Hadoop 1.x.
    
    i.e. -Dhadoopversion=20
    
    We need to move it under tez package.



src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
<https://reviews.apache.org/r/15194/#comment54641>

    This reverts PIG-3060 and breaks TestEvalPipelineLocal.testFlattenEmptyBag.
    
    https://issues.apache.org/jira/browse/PIG-3060


- Cheolsoo Park


On Nov. 2, 2013, 1:17 a.m., Mark Wagner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15194/
> -----------------------------------------------------------
> 
> (Updated Nov. 2, 2013, 1:17 a.m.)
> 
> 
> Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
> 
> 
> Bugs: PIG-3527
>     https://issues.apache.org/jira/browse/PIG-3527
> 
> 
> Repository: pig-git
> 
> 
> Description
> -------
> 
> Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
> * POSimpleTezLoad which consumes a single MRInput
> * POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
> The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).
> 
> 
> Diffs
> -----
> 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
>   src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
>   src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
>   src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
>   src/org/apache/pig/data/BinInterSedes.java b3ec51e 
>   src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
>   test/e2e/pig/tests/tez.conf 24af8d3 
> 
> Diff: https://reviews.apache.org/r/15194/diff/
> 
> 
> Testing
> -------
> 
> Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.
> 
> 
> Thanks,
> 
> Mark Wagner
> 
>


Re: Review Request 15194: Support multiple inputs for PigProcessor

Posted by Mark Wagner <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------

(Updated Nov. 2, 2013, 1:17 a.m.)


Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.


Bugs: PIG-3527
    https://issues.apache.org/jira/browse/PIG-3527


Repository: pig-git


Description
-------

Adds support for multiple LogicalInputs to the PigProcessor. This is done by adding a new TezLoad interface which PhysicalOperators may implement. On the backend, any operators implementing this interface will have the LogicalInput attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for the operator pipeline. This required a change to the comparators used so that the sort order remained consistent. There is also a fix to POForEach where it was using the incorrect status code for signaling (although it produced the same end result in the MR pipeline).


Diffs
-----

  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java ddea99e 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java 5ea3fc7 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java dfd4ebf 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java 09397e5 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java a87161f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java cbf457f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java 1d86e3f 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java bb6c9df 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java b3ded76 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java 5ad334b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java 022f37b 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java 866c39d 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java 9724b9f 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/TezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java eb9f62a 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java 86314d9 
  src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java c200715 
  src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java d29e330 
  src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java d2298ca 
  src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java ebb3145 
  src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java d7b42b8 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 45e47b0 
  src/org/apache/pig/data/BinInterSedes.java b3ec51e 
  src/org/apache/pig/data/DefaultTuple.java 2e7ca5f 
  test/e2e/pig/tests/tez.conf 24af8d3 

Diff: https://reviews.apache.org/r/15194/diff/


Testing
-------

Manual testing and an e2e test has been added. Because of the comparator change, some of the tests fail because of bag ordering.


Thanks,

Mark Wagner