[jira] Commented: (PIG-1062) load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface

["Thejas M Nair (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771615#action_12771615 ] 

Thejas M Nair commented on PIG-1062:
------------------------------------

Skew-join uses the total number of input tuples, in PartitionSkewedKeys.calculateReducers(..) to calculate number of reducers.
In the version in trunk, PoissonSampleLoader adds  size on disk of the sampled tuple , as the last column of the tuple. This is used to calculate average size on disk in PartitionSkewedKeys. Total number of tuples are estimated using input-file-size/avg-size-of-tuple-on-disk .

But with the new interface, the size on disk for a tuple cannot be estimated (there is no getPosition). Also, the size of input file on disk cannot be estimated if the input is not from a file or if the load function is passed some metadata instead of file name.

Ideally this information should be obtained through  ResourceStatistics in the proposal. Since that is not available right now, here is another proposal - 

PoissonSampleLoader currently reads almost all the rows because it tries to sample evenly spaced tuples from the split. It will now read till the last tuple, and add an additional tuple that has the number of tuples in that split. This special tuple needs to be distinguished from others that are sampled tuples. I don't have a good way to do that except for having two columns first column having an unique marker string, and second column has the number of rows. Does anybody have better suggestions ?

PartitionSkewedKeys will look at all these special rows and add the row-nums to get total number of rows.


> load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface 
> ---------------------------------------------------------------------------------------------------
>
>                 Key: PIG-1062
>                 URL: https://issues.apache.org/jira/browse/PIG-1062
>             Project: Pig
>          Issue Type: Task
>            Reporter: Thejas M Nair
>
> This is part of the effort to implement new load store interfaces as laid out in http://wiki.apache.org/pig/LoadStoreRedesignProposal .
> PigStorage and BinStorage are now working.
> SampleLoader and subclasses -RandomSampleLoader, PoissonSampleLoader need to be changed to work with new LoadFunc interface.  
> Fixing SampleLoader and RandomSampleLoader will get order-by queries working.
> PoissonSampleLoader is used by skew join. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.