You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by zhang jianfeng <zj...@gmail.com> on 2009/08/03 09:40:12 UTC
Is it possible to access Configuration in UDF ?
Hi all,
I'd like to set property in Configuration to customize my UDF. But it looks
like I can not access the Configuration object in UDF.
Does pig have a plan to support this feature ?
Thank you.
Jeff Zhang
RE: Is it possible to access Configuration in UDF ?
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
At the moment we can't make UDFs dependant on Hadoop as people also use
them for testing in local mode which is currently not based on Hadoop
local mode due to performance constrains.
I agree that we need to provide a way to get UDF a
configuration/property object.
Olga
-----Original Message-----
From: Daniel Dai [mailto:daijyc@gmail.com]
Sent: Monday, August 03, 2009 9:20 PM
To: pig-dev@hadoop.apache.org; pig-user@hadoop.apache.org
Subject: Re: Is it possible to access Configuration in UDF ?
Hi, Jeff,
This is not API at all, this is a hack to make things work. We do lack
couples of features for UDF:
1. reporter and counter (PIG-889)
2. access global properties
3. ability to maintain states across different UDF invocations
4. input schema
5. variable length arguments (PIG-902)
Your suggestion sounds resonable. We need to provide a well designed
interface for these features.
----- Original Message -----
From: "zhang jianfeng" <zj...@gmail.com>
To: <pi...@hadoop.apache.org>; <pi...@hadoop.apache.org>
Sent: Monday, August 03, 2009 8:03 PM
Subject: Re: Is it possible to access Configuration in UDF ?
> Dmitriy,
>
> Thank you for your help.
>
> I find this way of using API is not so intuitive , I recommend the
base
> class of UDF to implements the Configurable interface.
> Then each UDF can use the getConf() to get the Configuration object.
> Because UDF is part of MapReduce , it makes sense to make it
Configurable.
>
> The following is what I recommend to change the EvalFunc
>
> public abstract class EvalFunc<T> implements Configurable{
> ......
> protected Configuration conf;
> ......
> public EvalFunc(){
> conf=PigMapReduce.sJobConf;
> }
> ......
> @Override
> public void setConf(Configuration conf) {
> this.conf=conf;
> }
>
> @Override
> public Configuration getConf() {
> return this.conf;
> }
>
>
>
>
> Jeff Zhang
>
>
>
>
>
> On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy
> <dv...@cloudera.com>wrote:
>
>> You can access the JobConf with the following call:
>>
>> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>>
>> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com>
wrote:
>> > Hi all,
>> >
>> > I'd like to set property in Configuration to customize my UDF. But
it
>> looks
>> > like I can not access the Configuration object in UDF.
>> >
>> > Does pig have a plan to support this feature ?
>> >
>> >
>> > Thank you.
>> >
>> > Jeff Zhang
>> >
>>
>
RE: Is it possible to access Configuration in UDF ?
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
At the moment we can't make UDFs dependant on Hadoop as people also use
them for testing in local mode which is currently not based on Hadoop
local mode due to performance constrains.
I agree that we need to provide a way to get UDF a
configuration/property object.
Olga
-----Original Message-----
From: Daniel Dai [mailto:daijyc@gmail.com]
Sent: Monday, August 03, 2009 9:20 PM
To: pig-dev@hadoop.apache.org; pig-user@hadoop.apache.org
Subject: Re: Is it possible to access Configuration in UDF ?
Hi, Jeff,
This is not API at all, this is a hack to make things work. We do lack
couples of features for UDF:
1. reporter and counter (PIG-889)
2. access global properties
3. ability to maintain states across different UDF invocations
4. input schema
5. variable length arguments (PIG-902)
Your suggestion sounds resonable. We need to provide a well designed
interface for these features.
----- Original Message -----
From: "zhang jianfeng" <zj...@gmail.com>
To: <pi...@hadoop.apache.org>; <pi...@hadoop.apache.org>
Sent: Monday, August 03, 2009 8:03 PM
Subject: Re: Is it possible to access Configuration in UDF ?
> Dmitriy,
>
> Thank you for your help.
>
> I find this way of using API is not so intuitive , I recommend the
base
> class of UDF to implements the Configurable interface.
> Then each UDF can use the getConf() to get the Configuration object.
> Because UDF is part of MapReduce , it makes sense to make it
Configurable.
>
> The following is what I recommend to change the EvalFunc
>
> public abstract class EvalFunc<T> implements Configurable{
> ......
> protected Configuration conf;
> ......
> public EvalFunc(){
> conf=PigMapReduce.sJobConf;
> }
> ......
> @Override
> public void setConf(Configuration conf) {
> this.conf=conf;
> }
>
> @Override
> public Configuration getConf() {
> return this.conf;
> }
>
>
>
>
> Jeff Zhang
>
>
>
>
>
> On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy
> <dv...@cloudera.com>wrote:
>
>> You can access the JobConf with the following call:
>>
>> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>>
>> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com>
wrote:
>> > Hi all,
>> >
>> > I'd like to set property in Configuration to customize my UDF. But
it
>> looks
>> > like I can not access the Configuration object in UDF.
>> >
>> > Does pig have a plan to support this feature ?
>> >
>> >
>> > Thank you.
>> >
>> > Jeff Zhang
>> >
>>
>
Re: Is it possible to access Configuration in UDF ?
Posted by Daniel Dai <da...@gmail.com>.
Hi, Jeff,
This is not API at all, this is a hack to make things work. We do lack
couples of features for UDF:
1. reporter and counter (PIG-889)
2. access global properties
3. ability to maintain states across different UDF invocations
4. input schema
5. variable length arguments (PIG-902)
Your suggestion sounds resonable. We need to provide a well designed
interface for these features.
----- Original Message -----
From: "zhang jianfeng" <zj...@gmail.com>
To: <pi...@hadoop.apache.org>; <pi...@hadoop.apache.org>
Sent: Monday, August 03, 2009 8:03 PM
Subject: Re: Is it possible to access Configuration in UDF ?
> Dmitriy,
>
> Thank you for your help.
>
> I find this way of using API is not so intuitive , I recommend the base
> class of UDF to implements the Configurable interface.
> Then each UDF can use the getConf() to get the Configuration object.
> Because UDF is part of MapReduce , it makes sense to make it Configurable.
>
> The following is what I recommend to change the EvalFunc
>
> public abstract class EvalFunc<T> implements Configurable{
> ......
> protected Configuration conf;
> ......
> public EvalFunc(){
> conf=PigMapReduce.sJobConf;
> }
> ......
> @Override
> public void setConf(Configuration conf) {
> this.conf=conf;
> }
>
> @Override
> public Configuration getConf() {
> return this.conf;
> }
>
>
>
>
> Jeff Zhang
>
>
>
>
>
> On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy
> <dv...@cloudera.com>wrote:
>
>> You can access the JobConf with the following call:
>>
>> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>>
>> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com> wrote:
>> > Hi all,
>> >
>> > I'd like to set property in Configuration to customize my UDF. But it
>> looks
>> > like I can not access the Configuration object in UDF.
>> >
>> > Does pig have a plan to support this feature ?
>> >
>> >
>> > Thank you.
>> >
>> > Jeff Zhang
>> >
>>
>
Re: Is it possible to access Configuration in UDF ?
Posted by Daniel Dai <da...@gmail.com>.
Hi, Jeff,
This is not API at all, this is a hack to make things work. We do lack
couples of features for UDF:
1. reporter and counter (PIG-889)
2. access global properties
3. ability to maintain states across different UDF invocations
4. input schema
5. variable length arguments (PIG-902)
Your suggestion sounds resonable. We need to provide a well designed
interface for these features.
----- Original Message -----
From: "zhang jianfeng" <zj...@gmail.com>
To: <pi...@hadoop.apache.org>; <pi...@hadoop.apache.org>
Sent: Monday, August 03, 2009 8:03 PM
Subject: Re: Is it possible to access Configuration in UDF ?
> Dmitriy,
>
> Thank you for your help.
>
> I find this way of using API is not so intuitive , I recommend the base
> class of UDF to implements the Configurable interface.
> Then each UDF can use the getConf() to get the Configuration object.
> Because UDF is part of MapReduce , it makes sense to make it Configurable.
>
> The following is what I recommend to change the EvalFunc
>
> public abstract class EvalFunc<T> implements Configurable{
> ......
> protected Configuration conf;
> ......
> public EvalFunc(){
> conf=PigMapReduce.sJobConf;
> }
> ......
> @Override
> public void setConf(Configuration conf) {
> this.conf=conf;
> }
>
> @Override
> public Configuration getConf() {
> return this.conf;
> }
>
>
>
>
> Jeff Zhang
>
>
>
>
>
> On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy
> <dv...@cloudera.com>wrote:
>
>> You can access the JobConf with the following call:
>>
>> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>>
>> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com> wrote:
>> > Hi all,
>> >
>> > I'd like to set property in Configuration to customize my UDF. But it
>> looks
>> > like I can not access the Configuration object in UDF.
>> >
>> > Does pig have a plan to support this feature ?
>> >
>> >
>> > Thank you.
>> >
>> > Jeff Zhang
>> >
>>
>
Re: Is it possible to access Configuration in UDF ?
Posted by zhang jianfeng <zj...@gmail.com>.
Dmitriy,
Thank you for your help.
I find this way of using API is not so intuitive , I recommend the base
class of UDF to implements the Configurable interface.
Then each UDF can use the getConf() to get the Configuration object.
Because UDF is part of MapReduce , it makes sense to make it Configurable.
The following is what I recommend to change the EvalFunc
public abstract class EvalFunc<T> implements Configurable{
......
protected Configuration conf;
......
public EvalFunc(){
conf=PigMapReduce.sJobConf;
}
......
@Override
public void setConf(Configuration conf) {
this.conf=conf;
}
@Override
public Configuration getConf() {
return this.conf;
}
Jeff Zhang
On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> You can access the JobConf with the following call:
>
> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>
> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com> wrote:
> > Hi all,
> >
> > I'd like to set property in Configuration to customize my UDF. But it
> looks
> > like I can not access the Configuration object in UDF.
> >
> > Does pig have a plan to support this feature ?
> >
> >
> > Thank you.
> >
> > Jeff Zhang
> >
>
Re: Is it possible to access Configuration in UDF ?
Posted by zhang jianfeng <zj...@gmail.com>.
Dmitriy,
Thank you for your help.
I find this way of using API is not so intuitive , I recommend the base
class of UDF to implements the Configurable interface.
Then each UDF can use the getConf() to get the Configuration object.
Because UDF is part of MapReduce , it makes sense to make it Configurable.
The following is what I recommend to change the EvalFunc
public abstract class EvalFunc<T> implements Configurable{
......
protected Configuration conf;
......
public EvalFunc(){
conf=PigMapReduce.sJobConf;
}
......
@Override
public void setConf(Configuration conf) {
this.conf=conf;
}
@Override
public Configuration getConf() {
return this.conf;
}
Jeff Zhang
On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy <dv...@cloudera.com>wrote:
> You can access the JobConf with the following call:
>
> ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
>
> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com> wrote:
> > Hi all,
> >
> > I'd like to set property in Configuration to customize my UDF. But it
> looks
> > like I can not access the Configuration object in UDF.
> >
> > Does pig have a plan to support this feature ?
> >
> >
> > Thank you.
> >
> > Jeff Zhang
> >
>
Re: Is it possible to access Configuration in UDF ?
Posted by Dmitriy Ryaboy <dv...@cloudera.com>.
You can access the JobConf with the following call:
ConfigurationUtil.toProperties(PigMapReduce.sJobConf)
On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng<zj...@gmail.com> wrote:
> Hi all,
>
> I'd like to set property in Configuration to customize my UDF. But it looks
> like I can not access the Configuration object in UDF.
>
> Does pig have a plan to support this feature ?
>
>
> Thank you.
>
> Jeff Zhang
>