You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2014/02/04 18:40:39 UTC

Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/
-----------------------------------------------------------

Review request for pig, Cheolsoo Park and Daniel Dai.


Bugs: PIG-3742
    https://issues.apache.org/jira/browse/PIG-3742


Repository: pig


Description
-------

Changes made:
1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.


Diffs
-----

  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION 
  http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION 

Diff: https://reviews.apache.org/r/17681/diff/


Testing
-------

Unit and tez.conf e2e tests pass.


Thanks,

Rohini Palaniswamy

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Rohini Palaniswamy <ro...@gmail.com>.


> On Feb. 4, 2014, 8:05 p.m., Daniel Dai wrote:
> > Seems the patch cannot apply cleanly. Can you rebase?

This is because of the svn mv of the util classes. Uploaded a patch after deleting the moved files and creating them newly. Will do svn mv during the commit.


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/#review33632
-----------------------------------------------------------


On Feb. 4, 2014, 5:40 p.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17681/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2014, 5:40 p.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-3742
>     https://issues.apache.org/jira/browse/PIG-3742
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Changes made:
> 1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
> 2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
> 3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/17681/diff/
> 
> 
> Testing
> -------
> 
> Unit and tez.conf e2e tests pass.
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Daniel Dai <da...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/#review33632
-----------------------------------------------------------


Seems the patch cannot apply cleanly. Can you rebase?

- Daniel Dai


On Feb. 4, 2014, 5:40 p.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17681/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2014, 5:40 p.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-3742
>     https://issues.apache.org/jira/browse/PIG-3742
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Changes made:
> 1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
> 2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
> 3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/17681/diff/
> 
> 
> Testing
> -------
> 
> Unit and tez.conf e2e tests pass.
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Daniel Dai <da...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/#review33869
-----------------------------------------------------------

Ship it!


Ship It!

- Daniel Dai


On Feb. 6, 2014, 11:48 p.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17681/
> -----------------------------------------------------------
> 
> (Updated Feb. 6, 2014, 11:48 p.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-3742
>     https://issues.apache.org/jira/browse/PIG-3742
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Changes made:
> 1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
> 2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
> 3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/17681/diff/
> 
> 
> Testing
> -------
> 
> Unit and tez.conf e2e tests pass.
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Rohini Palaniswamy <ro...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/
-----------------------------------------------------------

(Updated Feb. 6, 2014, 11:48 p.m.)

Review request for pig, Cheolsoo Park and Daniel Dai.

Changes
-------

Updated patch. Had done some minor fixes. Also created tez staging dir in FileLocalizer relativeRoot directory.

Bugs: PIG-3742
https://issues.apache.org/jira/browse/PIG-3742

Repository: pig

Description
-------

Changes made:
1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.

Diffs (updated)
-----

http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION

Diff: https://reviews.apache.org/r/17681/diff/

Testing
-------

Unit and tez.conf e2e tests pass.

Thanks,

Rohini Palaniswamy

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Daniel Dai <da...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/#review33863
-----------------------------------------------------------

Ship it!


Ship It!

- Daniel Dai


On Feb. 4, 2014, 10:08 p.m., Rohini Palaniswamy wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17681/
> -----------------------------------------------------------
> 
> (Updated Feb. 4, 2014, 10:08 p.m.)
> 
> 
> Review request for pig, Cheolsoo Park and Daniel Dai.
> 
> 
> Bugs: PIG-3742
>     https://issues.apache.org/jira/browse/PIG-3742
> 
> 
> Repository: pig
> 
> 
> Description
> -------
> 
> Changes made:
> 1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
> 2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
> 3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION 
>   http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/17681/diff/
> 
> 
> Testing
> -------
> 
> Unit and tez.conf e2e tests pass.
> 
> 
> Thanks,
> 
> Rohini Palaniswamy
> 
>

Re: Review Request 17681: [PIG-3742] Set MR runtime settings on tez runtime

Posted by Rohini Palaniswamy <ro...@gmail.com>.

(Updated Feb. 4, 2014, 10:08 p.m.)

Review request for pig, Cheolsoo Park and Daniel Dai.

Changes
-------

Patch that deletes and newly adds util classes in util package instead of svn mv.

Bugs: PIG-3742
https://issues.apache.org/jira/browse/PIG-3742

Repository: pig

Description
-------

Diffs (updated)
-----

http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java 1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java PRE-CREATION

Diff: https://reviews.apache.org/r/17681/diff/

Testing
-------

Unit and tez.conf e2e tests pass.

Thanks,

Rohini Palaniswamy