You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/08/29 01:49:14 UTC
org.apache.hadoop.io.compress.SnappyCodec not found
Hi,
I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not found” error:
hadoop checknative
14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
zlib: true /lib64/libz.so.1
snappy: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
lz4: true revision:99
bzip2: false
(smoke test is ok)
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar teragen 300000 /tmp/teragenout
14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002
14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in uber mode : false
14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0%
14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0%
14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0%
14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed successfully
14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=197312
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=167
HDFS: Number of bytes written=30000000
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Job Counters
Launched map tasks=2
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=11925
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=11925
Total vcore-seconds taken by all map tasks=11925
Total megabyte-seconds taken by all map tasks=109900800
Map-Reduce Framework
Map input records=300000
Map output records=300000
Input split bytes=167
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=22
CPU time spent (ms)=1910
Physical memory (bytes) snapshot=357318656
Virtual memory (bytes) snapshot=1691631616
Total committed heap usage (bytes)=401997824
org.apache.hadoop.examples.terasort.TeraGen$Counters
CHECKSUM=644086318705578
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=30000000
14/08/29 07:41:03 INFO terasort.TeraSort: starting
14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process : 2
However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when running spark smoke test program:
scala> inFILE.first()
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.RDD.take(RDD.scala:983)
at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
at $iwC$$iwC$$iwC.<init>(<console>:20)
at $iwC$$iwC.<init>(<console>:22)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 55 more
Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 60 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
... 62 more
This is my setting in mapred-site.xml related to snappy
<property>
<name>mapred.output.compress</name>
<value>false</value>
<description>Should the job outputs be compressed?</description>
</property>
<property>
<name>mapred.output.compression.type</name>
<value>RECORD</value>
<description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK.</description>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
<description>If the job outputs are compressed, how should they be compressed?
</description>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
<description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression.</description>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
<description>If the map outputs are compressed, how should they be compressed?</description>
</property>
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
Can you please advise if my mapred-site.xml setting is incorrect?
Regards
Arthur
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
It looks a problem of class path at spark side.
Thanks,
- Tsuyoshi
On Fri, Aug 29, 2014 at 8:49 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not
> found” error:
>
> hadoop checknative
> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
> zlib: true /lib64/libz.so.1
> snappy: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
>
> (smoke test is ok)
> bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
> teragen 300000 /tmp/teragenout
> 14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002
> 14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in
> uber mode : false
> 14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0%
> 14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed
> successfully
> 14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=197312
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=167
> HDFS: Number of bytes written=30000000
> HDFS: Number of read operations=8
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=4
> Job Counters
> Launched map tasks=2
> Other local map tasks=2
> Total time spent by all maps in occupied slots (ms)=11925
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=11925
> Total vcore-seconds taken by all map tasks=11925
> Total megabyte-seconds taken by all map tasks=109900800
> Map-Reduce Framework
> Map input records=300000
> Map output records=300000
> Input split bytes=167
> Spilled Records=0
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=22
> CPU time spent (ms)=1910
> Physical memory (bytes) snapshot=357318656
> Virtual memory (bytes) snapshot=1691631616
> Total committed heap usage (bytes)=401997824
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=644086318705578
> File Input Format Counters
> Bytes Read=0
> File Output Format Counters
> Bytes Written=30000000
> 14/08/29 07:41:03 INFO terasort.TeraSort: starting
> 14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process :
> 2
>
>
> However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when
> running spark smoke test program:
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec
> org.apache.hadoop.io.compress.SnappyCodec not found.
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.io.compress.SnappyCodec not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
>
> This is my setting in mapred-site.xml related to snappy
> <property>
> <name>mapred.output.compress</name>
> <value>false</value>
> <description>Should the job outputs be compressed?</description>
> </property>
> <property>
> <name>mapred.output.compression.type</name>
> <value>RECORD</value>
> <description>If the job outputs are to compressed as SequenceFiles, how
> should they be compressed? Should be one of NONE, RECORD or
> BLOCK.</description>
> </property>
> <property>
> <name>mapred.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the job outputs are compressed, how should they be
> compressed?
> </description>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> <description>Should the outputs of the maps be compressed before being
> sent across the network. Uses SequenceFile compression.</description>
> </property>
> <property>
> <name>mapred.map.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the map outputs are compressed, how should they be
> compressed?</description>
> </property>
>
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.map.output.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
>
>
>
> Can you please advise if my mapred-site.xml setting is incorrect?
>
> Regards
> Arthur
>
--
- Tsuyoshi
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
It looks a problem of class path at spark side.
Thanks,
- Tsuyoshi
On Fri, Aug 29, 2014 at 8:49 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not
> found” error:
>
> hadoop checknative
> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
> zlib: true /lib64/libz.so.1
> snappy: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
>
> (smoke test is ok)
> bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
> teragen 300000 /tmp/teragenout
> 14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002
> 14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in
> uber mode : false
> 14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0%
> 14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed
> successfully
> 14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=197312
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=167
> HDFS: Number of bytes written=30000000
> HDFS: Number of read operations=8
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=4
> Job Counters
> Launched map tasks=2
> Other local map tasks=2
> Total time spent by all maps in occupied slots (ms)=11925
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=11925
> Total vcore-seconds taken by all map tasks=11925
> Total megabyte-seconds taken by all map tasks=109900800
> Map-Reduce Framework
> Map input records=300000
> Map output records=300000
> Input split bytes=167
> Spilled Records=0
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=22
> CPU time spent (ms)=1910
> Physical memory (bytes) snapshot=357318656
> Virtual memory (bytes) snapshot=1691631616
> Total committed heap usage (bytes)=401997824
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=644086318705578
> File Input Format Counters
> Bytes Read=0
> File Output Format Counters
> Bytes Written=30000000
> 14/08/29 07:41:03 INFO terasort.TeraSort: starting
> 14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process :
> 2
>
>
> However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when
> running spark smoke test program:
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec
> org.apache.hadoop.io.compress.SnappyCodec not found.
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.io.compress.SnappyCodec not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
>
> This is my setting in mapred-site.xml related to snappy
> <property>
> <name>mapred.output.compress</name>
> <value>false</value>
> <description>Should the job outputs be compressed?</description>
> </property>
> <property>
> <name>mapred.output.compression.type</name>
> <value>RECORD</value>
> <description>If the job outputs are to compressed as SequenceFiles, how
> should they be compressed? Should be one of NONE, RECORD or
> BLOCK.</description>
> </property>
> <property>
> <name>mapred.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the job outputs are compressed, how should they be
> compressed?
> </description>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> <description>Should the outputs of the maps be compressed before being
> sent across the network. Uses SequenceFile compression.</description>
> </property>
> <property>
> <name>mapred.map.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the map outputs are compressed, how should they be
> compressed?</description>
> </property>
>
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.map.output.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
>
>
>
> Can you please advise if my mapred-site.xml setting is incorrect?
>
> Regards
> Arthur
>
--
- Tsuyoshi
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
It looks a problem of class path at spark side.
Thanks,
- Tsuyoshi
On Fri, Aug 29, 2014 at 8:49 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not
> found” error:
>
> hadoop checknative
> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
> zlib: true /lib64/libz.so.1
> snappy: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
>
> (smoke test is ok)
> bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
> teragen 300000 /tmp/teragenout
> 14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002
> 14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in
> uber mode : false
> 14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0%
> 14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed
> successfully
> 14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=197312
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=167
> HDFS: Number of bytes written=30000000
> HDFS: Number of read operations=8
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=4
> Job Counters
> Launched map tasks=2
> Other local map tasks=2
> Total time spent by all maps in occupied slots (ms)=11925
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=11925
> Total vcore-seconds taken by all map tasks=11925
> Total megabyte-seconds taken by all map tasks=109900800
> Map-Reduce Framework
> Map input records=300000
> Map output records=300000
> Input split bytes=167
> Spilled Records=0
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=22
> CPU time spent (ms)=1910
> Physical memory (bytes) snapshot=357318656
> Virtual memory (bytes) snapshot=1691631616
> Total committed heap usage (bytes)=401997824
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=644086318705578
> File Input Format Counters
> Bytes Read=0
> File Output Format Counters
> Bytes Written=30000000
> 14/08/29 07:41:03 INFO terasort.TeraSort: starting
> 14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process :
> 2
>
>
> However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when
> running spark smoke test program:
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec
> org.apache.hadoop.io.compress.SnappyCodec not found.
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.io.compress.SnappyCodec not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
>
> This is my setting in mapred-site.xml related to snappy
> <property>
> <name>mapred.output.compress</name>
> <value>false</value>
> <description>Should the job outputs be compressed?</description>
> </property>
> <property>
> <name>mapred.output.compression.type</name>
> <value>RECORD</value>
> <description>If the job outputs are to compressed as SequenceFiles, how
> should they be compressed? Should be one of NONE, RECORD or
> BLOCK.</description>
> </property>
> <property>
> <name>mapred.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the job outputs are compressed, how should they be
> compressed?
> </description>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> <description>Should the outputs of the maps be compressed before being
> sent across the network. Uses SequenceFile compression.</description>
> </property>
> <property>
> <name>mapred.map.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the map outputs are compressed, how should they be
> compressed?</description>
> </property>
>
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.map.output.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
>
>
>
> Can you please advise if my mapred-site.xml setting is incorrect?
>
> Regards
> Arthur
>
--
- Tsuyoshi
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
It looks a problem of class path at spark side.
Thanks,
- Tsuyoshi
On Fri, Aug 29, 2014 at 8:49 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I use Hadoop 2.4.1, I got "org.apache.hadoop.io.compress.SnappyCodec not
> found” error:
>
> hadoop checknative
> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
> zlib: true /lib64/libz.so.1
> snappy: true
> /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
>
> (smoke test is ok)
> bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
> teragen 300000 /tmp/teragenout
> 14/08/29 07:40:41 INFO mapreduce.Job: Running job: job_1409253811850_0002
> 14/08/29 07:40:53 INFO mapreduce.Job: Job job_1409253811850_0002 running in
> uber mode : false
> 14/08/29 07:40:53 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/29 07:41:00 INFO mapreduce.Job: map 50% reduce 0%
> 14/08/29 07:41:01 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/29 07:41:02 INFO mapreduce.Job: Job job_1409253811850_0002 completed
> successfully
> 14/08/29 07:41:02 INFO mapreduce.Job: Counters: 31
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=197312
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=167
> HDFS: Number of bytes written=30000000
> HDFS: Number of read operations=8
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=4
> Job Counters
> Launched map tasks=2
> Other local map tasks=2
> Total time spent by all maps in occupied slots (ms)=11925
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=11925
> Total vcore-seconds taken by all map tasks=11925
> Total megabyte-seconds taken by all map tasks=109900800
> Map-Reduce Framework
> Map input records=300000
> Map output records=300000
> Input split bytes=167
> Spilled Records=0
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=22
> CPU time spent (ms)=1910
> Physical memory (bytes) snapshot=357318656
> Virtual memory (bytes) snapshot=1691631616
> Total committed heap usage (bytes)=401997824
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=644086318705578
> File Input Format Counters
> Bytes Read=0
> File Output Format Counters
> Bytes Written=30000000
> 14/08/29 07:41:03 INFO terasort.TeraSort: starting
> 14/08/29 07:41:03 INFO input.FileInputFormat: Total input paths to process :
> 2
>
>
> However I got "org.apache.hadoop.io.compress.SnappyCodec not found” when
> running spark smoke test program:
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec
> org.apache.hadoop.io.compress.SnappyCodec not found.
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at
> org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.io.compress.SnappyCodec not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at
> org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
>
> This is my setting in mapred-site.xml related to snappy
> <property>
> <name>mapred.output.compress</name>
> <value>false</value>
> <description>Should the job outputs be compressed?</description>
> </property>
> <property>
> <name>mapred.output.compression.type</name>
> <value>RECORD</value>
> <description>If the job outputs are to compressed as SequenceFiles, how
> should they be compressed? Should be one of NONE, RECORD or
> BLOCK.</description>
> </property>
> <property>
> <name>mapred.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the job outputs are compressed, how should they be
> compressed?
> </description>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> <description>Should the outputs of the maps be compressed before being
> sent across the network. Uses SequenceFile compression.</description>
> </property>
> <property>
> <name>mapred.map.output.compression.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> <description>If the map outputs are compressed, how should they be
> compressed?</description>
> </property>
>
> <property>
> <name>mapreduce.map.output.compress</name>
> <value>true</value>
> </property>
> <property>
> <name>mapred.map.output.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
>
>
>
> Can you please advise if my mapred-site.xml setting is incorrect?
>
> Regards
> Arthur
>
--
- Tsuyoshi