[jira] [Commented] (PIG-3500) Initial implementation of TezCompiler

["Mark Wagner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788761#comment-13788761 ] 

Mark Wagner commented on PIG-3500:
----------------------------------

Looks pretty good. The only major thing that stuck out is that for Tez, the special treatment of shuffles (Custom partitioners and combiners) needs to be associated with the inputs/outputs and not the TezOper since there can be many inputs/outputs. In the interest of progress on the Tez branch, I think it's best to commit this (with a couple small changes I noted and whatever others comment on) and handle the multi-combiner logic later. Do others have any problem with that?

> Initial implementation of TezCompiler
> -------------------------------------
>
>                 Key: PIG-3500
>                 URL: https://issues.apache.org/jira/browse/PIG-3500
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3500-1.patch, PIG-3500-2.patch
>
>
> Implement TezCompiler that compiles physical plan into tez plan. To begin with, we can implement the initial version that works for basic queries as follows:
> # Load-Filter-Store
> {code}
> a = load 'file:///tmp/input' as (x:int, y:int);
> b = filter a by x > 0;
> c = foreach b generate y;
> store c into 'file:///tmp/output';
> {code}
> # Load-Filter-GroupBy-Store
> {code}
> a = load 'file:///tmp/input' as (x:int, y:int);
> b = group a by x;
> c = foreach b generate group, a;
> store c into 'file:///tmp/output';
> {code}
> # Load1-Load2-Join-Store
> {code}
> a = load 'file:///tmp/input1' as (x:int, y:int);
> b = load 'file:///tmp/input2' as (x:int, z:int);
> c = join a by x, b by x;
> d = foreach c generate a::x as x, y, z;
> store d into 'file:///tmp/output';
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)