[jira] [Updated] (PIG-3742) Set MR runtime settings on tez runtime

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-3742:
------------------------------------

    Attachment: PIG-3742-3.patch

https://reviews.apache.org/r/17681

Changes made:
1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
4) Created tez staging dir in FileLocalizer relativeRoot directory so that it gets cleaned up.

> Set MR runtime settings on tez runtime
> --------------------------------------
>
>                 Key: PIG-3742
>                 URL: https://issues.apache.org/jira/browse/PIG-3742
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: tez-branch
>
>         Attachments: PIG-3742-3.patch
>
>
> Classify whether a vertex is a map or reduce and set java.opts (mapreduce.map.java.opts or mapreduce.reduce.java.opts), memory.mb (mapreduce.map.memory.mb or mapreduce.reduce.memory.mb) and env (mapreduce.map.env or mapreduce.reduce.env) accordingly on the vertex. A simple thing would be to assume all root vertexes to be map vertexes and intermediate or leaf vertexes to be reduce vertexes.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)