You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Alan Gates <ga...@yahoo-inc.com> on 2010/06/15 17:07:27 UTC

Re: Read HDFS from UDF?

FileLocalizer.openDFSFile() should do what you want.

Alan.

On May 28, 2010, at 3:04 PM, Mark Stetzer wrote:

> Hey all,
>
> I've seen this question asked on the mailing list in the past, but not
> recently.  Does anyone know of a way to read data from HDFS within a
> UDF?  I saw some discussion about a year ago that mentioned if a UDF
> implemented org.apache.hadoop.conf.Configurable it would get a copy of
> the Configuration object, but either that wasn't implemented, or I'm
> having troubles with it.
>
> Any thoughts?
>
> Thanks!
>
> -Mark

Re: Read HDFS from UDF?

Posted by Mark Stetzer <st...@gmail.com>.

Thanks.  I upgraded to Pig 0.6 (was using Cloudera's 0.5.x) and used
the UDFContext class, but this looks much more convenient.

-Mark

On Tue, Jun 15, 2010 at 11:07 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
> FileLocalizer.openDFSFile() should do what you want.
>
> Alan.
>
> On May 28, 2010, at 3:04 PM, Mark Stetzer wrote:
>
>> Hey all,
>>
>> I've seen this question asked on the mailing list in the past, but not
>> recently.  Does anyone know of a way to read data from HDFS within a
>> UDF?  I saw some discussion about a year ago that mentioned if a UDF
>> implemented org.apache.hadoop.conf.Configurable it would get a copy of
>> the Configuration object, but either that wasn't implemented, or I'm
>> having troubles with it.
>>
>> Any thoughts?
>>
>> Thanks!
>>
>> -Mark
>
>