[jira] [Commented] (TEZ-1003) Need a input that merges multiple ShuffleMergedInput from VertexGroup

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13968552#comment-13968552 ] 

Bikas Saha commented on TEZ-1003:
---------------------------------

FYI, This input does not have to be in the Tez code base. You can commit this to Pig to unblock your testing. Inputs classes are runtime dependencies and so all you need is a jar with this class in the classpath of the tasks. It can be any user jar.

> Need a input that merges multiple ShuffleMergedInput from VertexGroup
> ---------------------------------------------------------------------
>
>                 Key: TEZ-1003
>                 URL: https://issues.apache.org/jira/browse/TEZ-1003
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>         Attachments: TEZ-1003-1.patch
>
>
> In PIG-3835, was trying to do use vertex groups for unions. Union followed by store works fine. But when trying to do groupby, 
>  
> {code}
> A = LOAD '/tmp/data' AS (f1:int,f2:int);
> B = LOAD '/tmp/data2' AS (f1:int,f2:int);
> C = UNION onschema A,B;
> D = GROUP C by f1;
> E = FOREACH D GENERATE group, SUM(C.f2);
> store E into '/tmp/tezout' using PigStorage();
> {code}
> ConcatenatedMergedKeyValuesInput on the reduce, had only grouped records within each input and not across all inputs.
> i.e If A had records
> a 1
> b 1
> b 2
> and B
> a 2
> a 3
> b 3
> The records from ConcatenatedMergedKeyValuesInput of A and B were
> a {1}, b {1,2}, a {2,3}, b {3} while I am expecting a {1,2,3}, b {1,2,3}



--
This message was sent by Atlassian JIRA
(v6.2#6252)