You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by Dexin Wang <wa...@gmail.com> on 2010/11/24 02:25:47 UTC

pass configuration param to UDF

Hi all,

I was reading this:

http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs

It sounded like I can pass some configuration or context to the UDF but I
can't figure out how I would do that after I searched quite a bit on
internet and past discussion.

In my UDF, I can also do this:

                UDFContext context = UDFContext.getUDFContext();
                Properties properties =
context.getUDFProperties(this.getClass());

so if the context is set on the front end, supposedly, it will be in that
properties object. But how do I set it on the front end or whichever way to
pass it to UDF?

Thanks!
Dexin

Re: pass configuration param to UDF

Posted by Daniel Dai <ji...@yahoo-inc.com>.

You can pass -D using the command line:

java -Xmx512m -Dxxx=yyy -cp $HADOOP_CONF_DIR:pig.jar org.apache.pig.Main

Daniel

Dexin Wang wrote:
> Thanks Daniel!
>
> First option won't work for me since the property is not known to UDF
> itself, I need to pass it to UDF.
>
> Second option, I won't use pig.properties since the property passed to UDF
> are per pig script specific, not a global setting. How do I pass a -D option
> to pig script run (pig -f myscript.pig)?
>
> Thanks.
>
> On Tue, Nov 23, 2010 at 6:12 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
>
>   
>> The only hook in frontend for a UDF is outputSchema. You can put your
>> property into UDFContext in outputSchema, and read back in exec.
>>
>>   public String exec(Tuple input) throws IOException {
>>       UDFContext context = UDFContext.getUDFContext();
>>       String a =
>> context.getUDFProperties(this.getClass()).getProperty("Hello");
>>       return a;
>>   }
>>
>>   public Schema outputSchema(Schema input) {
>>       UDFContext context = UDFContext.getUDFContext();
>>       context.getUDFProperties(this.getClass()).setProperty("Hello",
>> "World");
>>       return null;
>>   }
>>
>> The other option is to provide a system wide configuration in command line
>> (-D), or pig.properties, which can be retrieved in UDF.exec using:
>> UDFContext.getUDFContext().getJobConf().get("propertyname")
>>
>> Daniel
>>
>>
>> Dexin Wang wrote:
>>
>>     
>>> Hi all,
>>>
>>> I was reading this:
>>>
>>> http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs
>>>
>>> It sounded like I can pass some configuration or context to the UDF but I
>>> can't figure out how I would do that after I searched quite a bit on
>>> internet and past discussion.
>>>
>>> In my UDF, I can also do this:
>>>
>>>                UDFContext context = UDFContext.getUDFContext();
>>>                Properties properties =
>>> context.getUDFProperties(this.getClass());
>>>
>>> so if the context is set on the front end, supposedly, it will be in that
>>> properties object. But how do I set it on the front end or whichever way
>>> to
>>> pass it to UDF?
>>>
>>> Thanks!
>>> Dexin
>>>
>>>
>>>       
>>

Re: pass configuration param to UDF

Posted by Dexin Wang <wa...@gmail.com>.

Thanks Daniel!

First option won't work for me since the property is not known to UDF
itself, I need to pass it to UDF.

Second option, I won't use pig.properties since the property passed to UDF
are per pig script specific, not a global setting. How do I pass a -D option
to pig script run (pig -f myscript.pig)?

Thanks.

On Tue, Nov 23, 2010 at 6:12 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:

> The only hook in frontend for a UDF is outputSchema. You can put your
> property into UDFContext in outputSchema, and read back in exec.
>
>   public String exec(Tuple input) throws IOException {
>       UDFContext context = UDFContext.getUDFContext();
>       String a =
> context.getUDFProperties(this.getClass()).getProperty("Hello");
>       return a;
>   }
>
>   public Schema outputSchema(Schema input) {
>       UDFContext context = UDFContext.getUDFContext();
>       context.getUDFProperties(this.getClass()).setProperty("Hello",
> "World");
>       return null;
>   }
>
> The other option is to provide a system wide configuration in command line
> (-D), or pig.properties, which can be retrieved in UDF.exec using:
> UDFContext.getUDFContext().getJobConf().get("propertyname")
>
> Daniel
>
>
> Dexin Wang wrote:
>
>> Hi all,
>>
>> I was reading this:
>>
>> http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs
>>
>> It sounded like I can pass some configuration or context to the UDF but I
>> can't figure out how I would do that after I searched quite a bit on
>> internet and past discussion.
>>
>> In my UDF, I can also do this:
>>
>>                UDFContext context = UDFContext.getUDFContext();
>>                Properties properties =
>> context.getUDFProperties(this.getClass());
>>
>> so if the context is set on the front end, supposedly, it will be in that
>> properties object. But how do I set it on the front end or whichever way
>> to
>> pass it to UDF?
>>
>> Thanks!
>> Dexin
>>
>>
>
>

Re: pass configuration param to UDF

Posted by Daniel Dai <ji...@yahoo-inc.com>.

The only hook in frontend for a UDF is outputSchema. You can put your 
property into UDFContext in outputSchema, and read back in exec.

    public String exec(Tuple input) throws IOException {
        UDFContext context = UDFContext.getUDFContext();
        String a = 
context.getUDFProperties(this.getClass()).getProperty("Hello");
        return a;
    }

    public Schema outputSchema(Schema input) {
        UDFContext context = UDFContext.getUDFContext();
        context.getUDFProperties(this.getClass()).setProperty("Hello", 
"World");
        return null;
    }

The other option is to provide a system wide configuration in command 
line (-D), or pig.properties, which can be retrieved in UDF.exec using:
UDFContext.getUDFContext().getJobConf().get("propertyname")

Daniel

Dexin Wang wrote:
> Hi all,
>
> I was reading this:
>
> http://pig.apache.org/docs/r0.7.0/udf.html#Passing+Configurations+to+UDFs
>
> It sounded like I can pass some configuration or context to the UDF but I
> can't figure out how I would do that after I searched quite a bit on
> internet and past discussion.
>
> In my UDF, I can also do this:
>
>                 UDFContext context = UDFContext.getUDFContext();
>                 Properties properties =
> context.getUDFProperties(this.getClass());
>
> so if the context is set on the front end, supposedly, it will be in that
> properties object. But how do I set it on the front end or whichever way to
> pass it to UDF?
>
> Thanks!
> Dexin
>