Using LoadFunc to get arbitrary data into Pig script

[Thomas Schlosser <th...@adtech.com>]

Hi all,
does anybody know what the easiest/best way is to get the result of an 
arbitrary calculation, which is done in Java, into a Pig relation?
At the end, I think I would like to have something like:
A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');

Note: Input file/directory is empty, since the input is configurable via 
external properties.

Since I have no distributed input, e.g., in the HDFS, the resulting Job 
should -- in my opinion -- run as a single map task.
Is there another mechanism in Pig or any special InputFormat that can be 
used?
I also had the problem that there is a mechanism that does not allow to 
have empty locations passed to the LoadFunc.
Any suggestions how to handle this?

Thanks in advance.

Regards,
Thomas

Re: Using LoadFunc to get arbitrary data into Pig script

[Dmitriy Ryaboy <dv...@gmail.com>]

Hi Thomas,
This isn't a complete answer, but take a look at mock.Storage that
Julien wrote to make testing easy:

http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/mock/Storage.java

D

On Fri, Sep 7, 2012 at 6:34 AM, Thomas Schlosser
<th...@adtech.com> wrote:
> Hi all,
> does anybody know what the easiest/best way is to get the result of an
> arbitrary calculation, which is done in Java, into a Pig relation?
> At the end, I think I would like to have something like:
> A = LOAD '' USING CustomCalculationLoadFunc('$paramA', '$paramB');
>
> Note: Input file/directory is empty, since the input is configurable via
> external properties.
>
> Since I have no distributed input, e.g., in the HDFS, the resulting Job
> should -- in my opinion -- run as a single map task.
> Is there another mechanism in Pig or any special InputFormat that can be
> used?
> I also had the problem that there is a mechanism that does not allow to have
> empty locations passed to the LoadFunc.
> Any suggestions how to handle this?
>
> Thanks in advance.
>
> Regards,
> Thomas