[jira] [Comment Edited] (PIG-3743) Use VertexGroup and Alias vertex for union

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948023#comment-13948023 ] 

Rohini Palaniswamy edited comment on PIG-3743 at 3/26/14 4:05 PM:
------------------------------------------------------------------

[~cheolsoo],
 
POLocalRearrangeTez.java    
{code}
if (isUnion) {
                        // Use the entire tuple as both key and value
                        key = HDataType.getWritableComparableTypes(result.get(1), keyType);
                        val = new NullableTuple((Tuple)result.get(1));
                    }
{code}

  This seems very inefficient as map output size just doubles and the key is just discarded. Can we use POValueOutputTez instead of POLocalRearrangeTez and use RoundRobinPartitioner instead.  Till now POValueOutputTez was only used with unordered input. Since the key is always POValueOutputTez.EmptyWritable, can you change it to implement WritableComparable and always return 1 (or -1 based on how comparator is called in reducer) so that no records are equal and they are not grouped together. If every key saying greater than other causes confusion while ordering and will not work, then we cannot use WritableComparable and have to go with each reducer having 1KV pair with all records grouped together. It might be overhead, but still should be better than having duplicating value in key and doubling the memory requirements.

Also will have to cleanup isUnion code in POLocalRearrangeTez if the solution works fine.


was (Author: rohini):
[~cheolsoo],
 
POLocalRearrangeTez.java    
{code}
if (isUnion) {
                        // Use the entire tuple as both key and value
                        key = HDataType.getWritableComparableTypes(result.get(1), keyType);
                        val = new NullableTuple((Tuple)result.get(1));
                    }
{code}

  This seems very inefficient as map output size just doubles and the key is just discarded. Can we use POValueOutputTez instead of POLocalRearrangeTez and use RoundRobinPartitioner instead.

> Use VertexGroup and Alias vertex for union
> ------------------------------------------
>
>                 Key: PIG-3743
>                 URL: https://issues.apache.org/jira/browse/PIG-3743
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>            Reporter: Rohini Palaniswamy
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3743-1.patch, PIG-3743-2.patch, PIG-3743-fix-skew.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)