[jira] Updated: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs

["Thejas M Nair (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thejas M Nair updated PIG-1472:
-------------------------------

    Attachment: PIG-1472.patch

Summary of changes in the patch -
1. The default TupleFactory is now BinSedesTupleFactory.  It returns the tuple implementation in BinSedesTuple class. This changes the serialization format between Map and Reduce.
2. The (de)serialization in BinSedesTuple and DefaultAbstractBag uses an implementation of a new InterSedes interface, which is returned by InterSedesFactory.getInterSedesInstance() 
3. A new load function InterStorage is used for serializing data between MR jobs . This load function should not be used like a regular load/store function to store persistent data.
4. DefaultTupleFactory has been retained, so that any external udfs that were using it can still compile. DefaultTupleFactory is a subclass of BinSedesTupleFactory that does not override any of the functions.

I think the serialization format should not be tied to the Tuple. A load function can return any tuple implementation, if we happen to call write function of that tuple, it will not be possible to read it in the reduce side using the default tuple. I think the InterSedes/InterSedesFactory classes should be used instead. With this patch, the InterSedes/InterSedesFactory classes get used only when BinSedesTuple is the default tuple.


> Optimize serialization/deserialization between Map and Reduce and between MR jobs
> ---------------------------------------------------------------------------------
>
>                 Key: PIG-1472
>                 URL: https://issues.apache.org/jira/browse/PIG-1472
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.8.0
>            Reporter: Thejas M Nair
>            Assignee: Thejas M Nair
>             Fix For: 0.8.0
>
>         Attachments: PIG-1472.patch
>
>
> In certain types of pig queries most of the execution time is spent in serializing/deserializing (sedes) records between Map and Reduce and between MR jobs. 
> For example, if PigMix queries are modified to specify types for all the fields in the load statement schema, some of the queries (L2,L3,L9, L10 in pigmix v1) that have records with bags and maps being transmitted across map or reduce boundaries run a lot longer (runtime increase of few times has been seen.
> There are a few optimizations that have shown to improve the performance of sedes in my tests -
> 1. Use smaller number of bytes to store length of the column . For example if a bytearray is smaller than 255 bytes , a byte can be used to store the length instead of the integer that is currently used.
> 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and DataInput.readUTF.  This reduces the cost of serialization by more than 1/2. 
> Zebra and BinStorage are known to use DefaultTuple sedes functionality. The serialization format that these loaders use cannot change, so after the optimization their format is going to be different from the format used between M/R boundaries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.