[jira] Commented: (PIG-1459) Need a standard way to communicate the requested fields between front and back end for loaders

["Alan Gates (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880293#action_12880293 ] 

Alan Gates commented on PIG-1459:
---------------------------------

>From email thread on pig-user:

I'm trying to figure out how exactly to appropriately implement the
LoadPushDown interface in my LoadFunc implementation. I need to take
the list of column aliases and pass that from the
LoadPushDown.pushProjection(RequiredFieldList) function to make it
available in the getTuple function. I'm kind of new to this so forgive
me if this is obvious. From my readings of the mailing list it appears
that the pushProjection function is called in the front-end where as
the getTuple function is called in the back-end. How does a LoanFunc
pass information from the front to the back end instances?

regards, Andrew

I wish there was better documentation on that too.

Looking at the PigStorage code, it serializes an array of Booleans via UDFContext to the backend.

It would be significantly better if Pig serialized the requested fields for us, provided that pushProjection returned a code that indicated that the projection would be supported.

Forcing users to do that serialization themselves is bug prone, especially in the presence of nested schemas.   



> Need a standard way to communicate the requested fields between front and back end for loaders
> ----------------------------------------------------------------------------------------------
>
>                 Key: PIG-1459
>                 URL: https://issues.apache.org/jira/browse/PIG-1459
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.7.0
>            Reporter: Alan Gates
>
> Pig currently provides no mechanism for loader writers to communicate which fields have been requested between the front and back end.  Since any loader that accepts pushed projections has to deal with this issue it would make sense for Pig to provide a standard mechanism for it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.