You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Andrew Look <al...@shopzilla.com> on 2011/03/26 01:50:07 UTC
Embedded Pig: Can hadoop cluster hostnames only be specified from
files in the classpath?
Hello Pig-Users,
I have been looking for a flexible way to specify the cluster hostnames of
the remote mapreduce instance that I want to connect to - I've looked
through the PigServer API / source code but haven't found a way of doing
this - it seems that if PigServer is instantiated in mapreduce mode, it only
looks in the classpath for core-site.xml & map-reduce-site.xml
I've found it very convenient to use code like the following for managing
data in HDFS using the Hadoop API:
Configuration config = new Configuration();
config.set("fs.default.name", hadoopClusterHdfsUrl);
fs = FileSystem.get(config);
Is there a way to do anything similar in embedded Pig? It seems that having
properties that can be filtered in at runtime would allow more flexibility
than depending on external files.
Thanks,
- Andrew
Re: Embedded Pig: Can hadoop cluster hostnames only be specified from
files in the classpath?
Posted by byambajav byambajargal <by...@gmail.com>.
Hello Andrew
I have a same problem as you have. if you find any solution please help me
as well.
Thanks
byambajargal
On Fri, Mar 25, 2011 at 8:50 PM, Andrew Look <al...@shopzilla.com> wrote:
> Hello Pig-Users,
>
> I have been looking for a flexible way to specify the cluster hostnames of
> the remote mapreduce instance that I want to connect to - I've looked
> through the PigServer API / source code but haven't found a way of doing
> this - it seems that if PigServer is instantiated in mapreduce mode, it
> only
> looks in the classpath for core-site.xml & map-reduce-site.xml
>
> I've found it very convenient to use code like the following for managing
> data in HDFS using the Hadoop API:
>
> Configuration config = new Configuration();
> config.set("fs.default.name", hadoopClusterHdfsUrl);
> fs = FileSystem.get(config);
>
> Is there a way to do anything similar in embedded Pig? It seems that having
> properties that can be filtered in at runtime would allow more flexibility
> than depending on external files.
>
> Thanks,
> - Andrew
>
>