You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Michael Sundell <ms...@gmail.com> on 2010/11/04 16:11:42 UTC
Passing Hadoop Site Configurations in classpath is not recommended
for Local Mode
Hi,
While trying to run Pig in local mode using "pig -x local" I get the
following error:
10/11/04 07:29:18 INFO pig.Main: Logging error messages to:
/Users/some-name/Software/cloudera/hadoop-0.20.2+737/pig_1288880958619.log
2010-11-04 07:29:18,807 [main] WARN
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
Hadoop Site Configurations in classpath is not recommended for Local Mode
2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
mapred.system.dir:
/Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
mentioned in the configuration does not exist
I found the following patch that talks about this issue and about a patch
that generates this error message:
https://issues.apache.org/jira/browse/PIG-1154
What I'm not clear about is how to address this issue.
I'm using:
pig-0.7.0+16
hadoop-0.20.2+737
Thank you
Re: Passing Hadoop Site Configurations in classpath is not recommended
for Local Mode
Posted by Daniel Dai <ji...@yahoo-inc.com>.
This is a defect in Pig 0.7. Pig 0.8 will automatically exclude hadoop
config file in local mode (https://issues.apache.org/jira/browse/PIG-1338)
Daniel
Michael Sundell wrote:
> It turns out that Pig calls $HADOOP_HOME/bin/hadoop-config.sh
> Inside this script this is set by default (among other things) which
> causes the warning:
>
> HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}"
>
> Since this is a default setting for a Hadoop installation I'm assuming
> that this warning can be ignored.
>
> /Michael
>
>
>> Looks like one of the environment variables in the pig shell script (`which pig`) is pointing to the hadoop cluster configuration. Check if environment variables PIG_CLASSPATH or PIG_CONF_DIR points to the hadoop configuration dir/file .
>> -Thejas
>>
>> Hi,
>>
>> While trying to run Pig in local mode using "pig -x local" I get the
>> following error:
>> 10/11/04 07:29:18 INFO pig.Main: Logging error messages to:
>> /Users/some-name/Software/cloudera/hadoop-0.20.2+737/pig_1288880958619.log
>> 2010-11-04 07:29:18,807 [main] WARN
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
>> Hadoop Site Configurations in classpath is not recommended for Local Mode
>> 2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
>> mapred.system.dir:
>> /Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
>> mentioned in the configuration does not exist
>>
>> I found the following patch that talks about this issue and about a patch
>> that generates this error message:
>> https://issues.apache.org/jira/browse/PIG-1154
>>
>> What I'm not clear about is how to address this issue.
>>
>> I'm using:
>> pig-0.7.0+16
>> hadoop-0.20.2+737
>>
>> Thank you
>>
>>
>>
Re: Passing Hadoop Site Configurations in classpath is not
recommended for Local Mode
Posted by Michael Sundell <ms...@gmail.com>.
It turns out that Pig calls $HADOOP_HOME/bin/hadoop-config.sh
Inside this script this is set by default (among other things) which
causes the warning:
HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}"
Since this is a default setting for a Hadoop installation I'm assuming
that this warning can be ignored.
/Michael
> Looks like one of the environment variables in the pig shell script (`which pig`) is pointing to the hadoop cluster configuration. Check if environment variables PIG_CLASSPATH or PIG_CONF_DIR points to the hadoop configuration dir/file .
> -Thejas
>
> Hi,
>
> While trying to run Pig in local mode using "pig -x local" I get the
> following error:
> 10/11/04 07:29:18 INFO pig.Main: Logging error messages to:
> /Users/some-name/Software/cloudera/hadoop-0.20.2+737/pig_1288880958619.log
> 2010-11-04 07:29:18,807 [main] WARN
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
> Hadoop Site Configurations in classpath is not recommended for Local Mode
> 2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
> mapred.system.dir:
> /Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
> mentioned in the configuration does not exist
>
> I found the following patch that talks about this issue and about a patch
> that generates this error message:
> https://issues.apache.org/jira/browse/PIG-1154
>
> What I'm not clear about is how to address this issue.
>
> I'm using:
> pig-0.7.0+16
> hadoop-0.20.2+737
>
> Thank you
>
>
Re: Passing Hadoop Site Configurations in classpath is not
recommended for Local Mode
Posted by Thejas M Nair <te...@yahoo-inc.com>.
Looks like one of the environment variables in the pig shell script (`which pig`) is pointing to the hadoop cluster configuration. Check if environment variables PIG_CLASSPATH or PIG_CONF_DIR points to the hadoop configuration dir/file .
-Thejas
On 11/4/10 8:11 AM, "Michael Sundell" <ms...@gmail.com> wrote:
Hi,
While trying to run Pig in local mode using "pig -x local" I get the
following error:
10/11/04 07:29:18 INFO pig.Main: Logging error messages to:
/Users/some-name/Software/cloudera/hadoop-0.20.2+737/pig_1288880958619.log
2010-11-04 07:29:18,807 [main] WARN
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
Hadoop Site Configurations in classpath is not recommended for Local Mode
2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
mapred.system.dir:
/Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
mentioned in the configuration does not exist
I found the following patch that talks about this issue and about a patch
that generates this error message:
https://issues.apache.org/jira/browse/PIG-1154
What I'm not clear about is how to address this issue.
I'm using:
pig-0.7.0+16
hadoop-0.20.2+737
Thank you
Re: Passing Hadoop Site Configurations in classpath is not
recommended for Local Mode
Posted by Rekha Joshi <re...@yahoo-inc.com>.
Hi,
AFAIK, the characteristic of the parameter mapred.system.dir is not conducive to be modified per job.It must be modified on mapred-default.xml, conf/mapred-site.xml.
So why local, I don't recall passing it on non-local mode either, and if passed I don't think it should work, as its final.
The parameters passed on command line usually pertain to the job performance/memory needs of the specific job.Makes sense?
Thanks & Regards,
/Rekha.
On 11/4/10 8:41 PM, "Michael Sundell" <ms...@gmail.com> wrote:
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
Hadoop Site Configurations in classpath is not recommended for Local Mode
2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
mapred.system.dir:
/Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
mentioned in the configuration does not exist
I found the following patch that talks about this issue and about