You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by rakesh sharma <ra...@hotmail.com> on 2012/02/06 23:56:00 UTC
Setting HDFS configuration in PigServer
Hi All,
I am using PigServer to execute pig scripts. It picks up the HDFS configuration from hadoop configuration files residing in the classpath. I am wondering if there is a way to pass these values to PigServer and eliminate the need of having HDFS configuration files.
Thanks,Rakesh
RE: Setting HDFS configuration in PigServer
Posted by rakesh sharma <ra...@hotmail.com>.
Hi Prashant,
I needed a way to control the hadoop configuration based on the deployed environment. I found a work-around by putting empty core-site.xml in the classpath and providing properties in the constructor.
Thanks for your help!
-Rakesh
> Date: Mon, 6 Feb 2012 16:27:03 -0800
> Subject: Re: Setting HDFS configuration in PigServer
> From: prash1784@gmail.com
> To: user@pig.apache.org
>
> I think I jumped the gun here, I was under the impression you want to
> override certain properties.
>
> In MapReduce mode, Pig always looks for "hadoop-site.xml" or
> "core-site.xml". I am not aware of anyway to bypass this.
>
> Curious, why do you want to not provide the config files?
>
> Thanks,
> Prashant
>
> On Mon, Feb 6, 2012 at 4:17 PM, rakesh sharma
> <ra...@hotmail.com>wrote:
>
> >
> > Hi Prashant,
> > I tried it before posting the question. Here is the exception that I get
> > when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note
> > that properties has the value of property "fs.default.name":
> > org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot
> > find hadoop configurations in classpath (neither hadoop-site.xml nor
> > core-site.xml was found in the classpath).If you plan to use local mode,
> > please put -x local option in command line at
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169)
> > at
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
> > at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
> > at org.apache.pig.PigServer.<init>(PigServer.java:226) at
> > org.apache.pig.PigServer.<init>(PigServer.java:215) at
> > org.apache.pig.PigServer.<init>(PigServer.java:211)
> > Any suggestions?
> > Thanks,Rakesh
> >
> > > Date: Mon, 6 Feb 2012 15:00:24 -0800
> > > Subject: Re: Setting HDFS configuration in PigServer
> > > From: prash1784@gmail.com
> > > To: user@pig.apache.org
> > >
> > > Rakesh,
> > >
> > > You could use
> > >
> > http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
> > >
> > > You can specify your own properties via this constructor.
> > >
> > > Something like,
> > >
> > > Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> > > you want to load defaults
> > > properties.set("foo","bar");
> > > PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
> > >
> > >
> > > Thanks,
> > > Prashant
> > >
> > > On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> > > <ra...@hotmail.com>wrote:
> > >
> > > >
> > > > Hi All,
> > > > I am using PigServer to execute pig scripts. It picks up the HDFS
> > > > configuration from hadoop configuration files residing in the
> > classpath. I
> > > > am wondering if there is a way to pass these values to PigServer and
> > > > eliminate the need of having HDFS configuration files.
> > > > Thanks,Rakesh
> >
> >
Re: Setting HDFS configuration in PigServer
Posted by Prashant Kommireddi <pr...@gmail.com>.
I think I jumped the gun here, I was under the impression you want to
override certain properties.
In MapReduce mode, Pig always looks for "hadoop-site.xml" or
"core-site.xml". I am not aware of anyway to bypass this.
Curious, why do you want to not provide the config files?
Thanks,
Prashant
On Mon, Feb 6, 2012 at 4:17 PM, rakesh sharma
<ra...@hotmail.com>wrote:
>
> Hi Prashant,
> I tried it before posting the question. Here is the exception that I get
> when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note
> that properties has the value of property "fs.default.name":
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot
> find hadoop configurations in classpath (neither hadoop-site.xml nor
> core-site.xml was found in the classpath).If you plan to use local mode,
> please put -x local option in command line at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
> at org.apache.pig.PigServer.<init>(PigServer.java:226) at
> org.apache.pig.PigServer.<init>(PigServer.java:215) at
> org.apache.pig.PigServer.<init>(PigServer.java:211)
> Any suggestions?
> Thanks,Rakesh
>
> > Date: Mon, 6 Feb 2012 15:00:24 -0800
> > Subject: Re: Setting HDFS configuration in PigServer
> > From: prash1784@gmail.com
> > To: user@pig.apache.org
> >
> > Rakesh,
> >
> > You could use
> >
> http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
> >
> > You can specify your own properties via this constructor.
> >
> > Something like,
> >
> > Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> > you want to load defaults
> > properties.set("foo","bar");
> > PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
> >
> >
> > Thanks,
> > Prashant
> >
> > On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> > <ra...@hotmail.com>wrote:
> >
> > >
> > > Hi All,
> > > I am using PigServer to execute pig scripts. It picks up the HDFS
> > > configuration from hadoop configuration files residing in the
> classpath. I
> > > am wondering if there is a way to pass these values to PigServer and
> > > eliminate the need of having HDFS configuration files.
> > > Thanks,Rakesh
>
>
RE: Setting HDFS configuration in PigServer
Posted by rakesh sharma <ra...@hotmail.com>.
Hi Prashant,
I tried it before posting the question. Here is the exception that I get when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note that properties has the value of property "fs.default.name":
org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).If you plan to use local mode, please put -x local option in command line at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.<init>(PigServer.java:226) at org.apache.pig.PigServer.<init>(PigServer.java:215) at org.apache.pig.PigServer.<init>(PigServer.java:211)
Any suggestions?
Thanks,Rakesh
> Date: Mon, 6 Feb 2012 15:00:24 -0800
> Subject: Re: Setting HDFS configuration in PigServer
> From: prash1784@gmail.com
> To: user@pig.apache.org
>
> Rakesh,
>
> You could use
> http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
>
> You can specify your own properties via this constructor.
>
> Something like,
>
> Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> you want to load defaults
> properties.set("foo","bar");
> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>
>
> Thanks,
> Prashant
>
> On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> <ra...@hotmail.com>wrote:
>
> >
> > Hi All,
> > I am using PigServer to execute pig scripts. It picks up the HDFS
> > configuration from hadoop configuration files residing in the classpath. I
> > am wondering if there is a way to pass these values to PigServer and
> > eliminate the need of having HDFS configuration files.
> > Thanks,Rakesh
Re: Setting HDFS configuration in PigServer
Posted by Prashant Kommireddi <pr...@gmail.com>.
Rakesh,
You could use
http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
You can specify your own properties via this constructor.
Something like,
Properties properties = PropertiesUtil.loadDefaultProperties(); //If
you want to load defaults
properties.set("foo","bar");
PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
Thanks,
Prashant
On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
<ra...@hotmail.com>wrote:
>
> Hi All,
> I am using PigServer to execute pig scripts. It picks up the HDFS
> configuration from hadoop configuration files residing in the classpath. I
> am wondering if there is a way to pass these values to PigServer and
> eliminate the need of having HDFS configuration files.
> Thanks,Rakesh