You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Tom Bryant <ma...@verizon.net> on 2013/10/27 14:14:27 UTC
NewBee Pig Problems.
I've downloaded pig and am trying to get it to do the simplest things,
following the Apache Getting Started instructions:
The download and installation seemed to go fine. Pig -help, from
the newly created pig account, produced a list of helpful commands.
Then the problems began:
pig -x local vomits out:
2013-10-27 09:02:27,759 [main] INFO org.apache.pig.Main - Apache Pig
version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2013-10-27 09:02:27,760 [main] INFO org.apache.pig.Main - Logging error
messages to: /science/hadoop/pig/pig_1382878947757.log
2013-10-27 09:02:27,794 [main] INFO org.apache.pig.impl.util.Utils -
Default bootup file /science/hadoop/pig/.pigbootup not found
2013-10-27 09:02:28,152 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: file:///
2013-10-27 09:02:28,157 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.used.genericoptionsparser is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used
2013-10-27 09:02:28,161 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2013-10-27 09:02:28,161 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address
2013-10-27 09:02:28,821 [main] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-10-27 09:02:28,964 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum
is deprecated. Instead, use dfs.bytes-per-checksum
2013-10-27 09:02:28,968 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
grunt>
Well, it seems to be running, so lets try it. As per the instructions,
I copied /etc/passwd to the pwd and then:
grunt> A = load 'passwd' using PigStorage(':');
2013-10-27 09:04:37,525 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
job.end.retry.interval is deprecated. Instead, use
mapreduce.job.end-noti ication.retry.interval
2013-10-27 09:04:37,525 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use
map educe.jobtracker.retiredjobs.cache.size
2013-10-27 09:04:37,525 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
mapred.task.profile.reduces is deprecated. Instead, use
mapreduce.task.pr file.reduces
2013-10-27 09:04:37,525 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use
mapreduce.job. vm.numtasks
...<About 50 lines of SNIP!>
2013-10-27 09:04:37,551 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion - hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
2013-10-27 09:04:37,552 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
mapreduce.job.counters.limit is deprecated. Instead, use
mapreduce.job.co nters.max
2013-10-27 09:04:37,552 [main] INFO
org.apache.hadoop.conf.Configuration.deprec tion -
mapred.job.reduce.input.buffer.percent is deprecated. Instead, use
mapred ce.reduce.input.buffer.percent
Not what I had expected.
If you've plodded along this far, and would like to take a stab at what
I'm doing wrong, TIA!
You'll need my environment:
Slackware linux 13.0.
uname -a gives:
Linux nebula 2.6.33.4 #3 SMP Wed May 12 23:13:09 CDT 2010 x86_64
Intel(R) Core(TM)2 Quad CPU Q8200 @ 2.33GHz GenuineIntel GNU/Linux
$PATH:
/usr/local/bin/
/usr/bin
/bin
/science/hadoop/pig/bin
/science/hadoop/pig/pig-0.11.1/bin
/science/hadoop/bin
/usr/local/lib64/jdk1.7.0_25
/usr/local/lib64/jdk1.7.0_25/bin
/usr/local/lib64/jdk1.7.0_25/visualvm_134/bin
/usr/local/apache-maven/bin
/work/glxy/bin
/work/python
/usr/local/lib64/apache-ant-1.9.2/bin
$CLASSPATH:
/usr/local/lib64/jdk1.7.0_25
/usr/local/lib64/jdk1.7.0_25/bin/
/usr/local/lib64/jdk1.7.0_25/lib/junit-4.11.jar
/usr/local/lib64/apache-ant-1.9.2/bin
/science/hadoop/pig/pig-0.11.1/pig-0.11.1.jar
/science/hadoop/pig/classes
I've tried pig-0.12.0 as well, with similar results.
Thanks,
Tom
Re: NewBee Pig Problems.
Posted by Tom Wheeler <to...@gmail.com>.
Despite all the messages emitted by Pig, you're not actually doing anything
wrong.
The issue is that Pig uses Hadoop, and Hadoop changed the names of some
configuration properties some time ago. Pig is using the older deprecated
names, as indicated in the messages you saw. This leads to the next logical
question...why doesn't Pig use the new names (and thus eliminate these
warnings)? Although there are others on this list who could answer that
with authority, I think it's because doing so would make Pig incompatible
with older versions of Hadoop (i.e. those that don't understand the new
property names).
So, the bottom line is that you can safely ignore those deprecation
warnings. You can suppress them by modifying the log4j configuration that
Pig uses. I'm pretty sure the steps for doing this have been mentioned on a
previous thread in this list, but write back and ask for details if you
can't find it.
On Sun, Oct 27, 2013 at 8:14 AM, Tom Bryant <ma...@verizon.net>wrote:
> I've downloaded pig and am trying to get it to do the simplest things,
> following the Apache Getting Started instructions:
>
> The download and installation seemed to go fine. Pig -help, from the
> newly created pig account, produced a list of helpful commands.
>
> Then the problems began:
>
> pig -x local vomits out:
>
> 2013-10-27 09:02:27,759 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
> 2013-10-27 09:02:27,760 [main] INFO org.apache.pig.Main - Logging error
> messages to: /science/hadoop/pig/pig_**1382878947757.log
> 2013-10-27 09:02:27,794 [main] INFO org.apache.pig.impl.util.Utils -
> Default bootup file /science/hadoop/pig/.pigbootup not found
> 2013-10-27 09:02:28,152 [main] INFO org.apache.pig.backend.hadoop.**
> executionengine.**HExecutionEngine - Connecting to hadoop file system at:
> file:///
> 2013-10-27 09:02:28,157 [main] INFO org.apache.hadoop.conf.**Configuration.deprecation
> - mapred.used.**genericoptionsparser is deprecated. Instead, use
> mapreduce.client.**genericoptionsparser.used
> 2013-10-27 09:02:28,161 [main] INFO org.apache.hadoop.conf.**Configuration.deprecation
> - fs.default.name is deprecated. Instead, use fs.defaultFS
> 2013-10-27 09:02:28,161 [main] INFO org.apache.hadoop.conf.**Configuration.deprecation
> - mapred.job.tracker is deprecated. Instead, use
> mapreduce.jobtracker.address
> 2013-10-27 09:02:28,821 [main] WARN org.apache.hadoop.util.**NativeCodeLoader
> - Unable to load native-hadoop library for your platform... using
> builtin-java classes where applicable
> 2013-10-27 09:02:28,964 [main] INFO org.apache.hadoop.conf.**Configuration.deprecation
> - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> 2013-10-27 09:02:28,968 [main] INFO org.apache.hadoop.conf.**Configuration.deprecation
> - fs.default.name is deprecated. Instead, use fs.defaultFS
> grunt>
>
> Well, it seems to be running, so lets try it. As per the instructions, I
> copied /etc/passwd to the pwd and then:
>
> grunt> A = load 'passwd' using PigStorage(':');
>
> 2013-10-27 09:04:37,525 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - job.end.retry.interval is deprecated. Instead, use
> mapreduce.job.end-noti ication.retry.interval
> 2013-10-27 09:04:37,525 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - mapred.job.tracker.**retiredjobs.cache.size is deprecated.
> Instead, use map educe.jobtracker.retiredjobs.**cache.size
> 2013-10-27 09:04:37,525 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - mapred.task.profile.reduces is deprecated. Instead, use
> mapreduce.task.pr file.reduces
> 2013-10-27 09:04:37,525 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use
> mapreduce.job. vm.numtasks
> ...<About 50 lines of SNIP!>
> 2013-10-27 09:04:37,551 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2013-10-27 09:04:37,552 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - mapreduce.job.counters.limit is deprecated. Instead, use
> mapreduce.job.co nters.max
> 2013-10-27 09:04:37,552 [main] INFO org.apache.hadoop.conf.**Configuration.deprec
> tion - mapred.job.reduce.input.**buffer.percent is deprecated. Instead,
> use mapred ce.reduce.input.buffer.percent
>
> Not what I had expected.
>
> If you've plodded along this far, and would like to take a stab at what
> I'm doing wrong, TIA!
>
> You'll need my environment:
>
> Slackware linux 13.0.
>
> uname -a gives:
>
> Linux nebula 2.6.33.4 #3 SMP Wed May 12 23:13:09 CDT 2010 x86_64 Intel(R)
> Core(TM)2 Quad CPU Q8200 @ 2.33GHz GenuineIntel GNU/Linux
>
> $PATH:
> /usr/local/bin/
> /usr/bin
> /bin
> /science/hadoop/pig/bin
> /science/hadoop/pig/pig-0.11.**1/bin
> /science/hadoop/bin
> /usr/local/lib64/jdk1.7.0_25
> /usr/local/lib64/jdk1.7.0_25/**bin
> /usr/local/lib64/jdk1.7.0_25/**visualvm_134/bin
> /usr/local/apache-maven/bin
> /work/glxy/bin
> /work/python
> /usr/local/lib64/apache-ant-1.**9.2/bin
>
> $CLASSPATH:
> /usr/local/lib64/jdk1.7.0_25
> /usr/local/lib64/jdk1.7.0_25/**bin/
> /usr/local/lib64/jdk1.7.0_25/**lib/junit-4.11.jar
> /usr/local/lib64/apache-ant-1.**9.2/bin
> /science/hadoop/pig/pig-0.11.**1/pig-0.11.1.jar
> /science/hadoop/pig/classes
>
> I've tried pig-0.12.0 as well, with similar results.
>
> Thanks,
>
> Tom
>
>
>
--
Tom Wheeler
http://www.tomwheeler.com/