You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "yingnan.ma" <yi...@ipinyou.com> on 2012/06/11 15:56:20 UTC
lzo error
Hi
I already install the lzo, and from the log it sould like successful
mapred.compress.map.output true
&&
syslog logs
2012-06-11 21:50:39,749 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-06-11 21:50:40,247 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2012-06-11 21:50:40,352 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring.
2012-06-11 21:50:40,361 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring.
2012-06-11 21:50:41,118 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 150
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: data buffer = 119537664/149422080
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: record buffer = 393216/491520
2012-06-11 21:50:41,276 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2012-06-11 21:50:41,280 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
but the pig will give me an error
2012-06-11 21:50:31,507 [Thread-4] ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl libraryjava.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738) at java.lang.Runtime.loadLibrary0(Runtime.java:823) at java.lang.System.loadLibrary(System.java:1028) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156) at org.apache.hadoop.mapreduce.lib.input.
TextInputFormat.isSplitable(TextInputFormat.java:51) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) at org.apache.hadoop.mapred.JobCl
ient.submitJob(JobClient.java:807) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:662)2012-06-11 21:50:31,508 [Thread-4] ERROR com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without native-hadoopAnd the job could run successful, I donot know the reason. Thank you for your help.Best RegardsMalone
2012-06-11
Re: lzo error
Posted by "yingnan.ma" <yi...@ipinyou.com>.
sorry,I already got the problem and address it, sorry to trouble with you!
Best Regards
Malone
2012-06-11
yingnan.ma
发件人： yingnan.ma
发送时间： 2012-06-11 21:57:15
收件人： user
抄送：
主题： lzo error
Hi
I already install the lzo, and from the log it sould like successful
mapred.compress.map.output true
&&
syslog logs
2012-06-11 21:50:39,749 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-06-11 21:50:40,247 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2012-06-11 21:50:40,352 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring.
2012-06-11 21:50:40,361 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring.
2012-06-11 21:50:41,118 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 150
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: data buffer = 119537664/149422080
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: record buffer = 393216/491520
2012-06-11 21:50:41,276 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2012-06-11 21:50:41,280 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
but the pig will give me an error
2012-06-11 21:50:31,507 [Thread-4] ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl libraryjava.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738) at java.lang.Runtime.loadLibrary0(Runtime.java:823) at java.lang.System.loadLibrary(System.java:1028) at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32) at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156) at org.apache.hadoop.mapreduce.lib.input.
TextInputFormat.isSplitable(TextInputFormat.java:51) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) at org.apache.hadoop.mapred.JobCl
ient.submitJob(JobClient.java:807) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:662)2012-06-11 21:50:31,508 [Thread-4] ERROR com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without native-hadoopAnd the job could run successful, I donot know the reason. Thank you for your help.Best RegardsMalone
2012-06-11