You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/08/28 18:50:56 UTC
SPARK-1297 patch error (spark-1297-v4.txt )
Hi,
I have just tried to apply the patch of SPARK-1297:
https://issues.apache.org/jira/browse/SPARK-1297
There are two files in it, named spark-1297-v2.txt and spark-1297-v4.txt respectively.
When applying the 2nd one, I got "Hunk #1 FAILED at 45"
Can you please advise how to fix it in order to make the compilation of Spark Project Examples success?
(Here: Hadoop 2.4.1, HBase 0.98.5, Spark 1.0.2)
Regards
Arthur
patch -p1 -i spark-1297-v4.txt
patching file examples/pom.xml
Hunk #1 FAILED at 45.
Hunk #2 succeeded at 94 (offset -16 lines).
1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
below is the content of examples/pom.xml.rej:
+++ examples/pom.xml
@@ -45,6 +45,39 @@
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>hbase-hadoop2</id>
+ <activation>
+ <property>
+ <name>hbase.profile</name>
+ <value>hadoop2</value>
+ </property>
+ </activation>
+ <properties>
+ <protobuf.version>2.5.0</protobuf.version>
+ <hbase.version>0.98.4-hadoop2</hbase.version>
+ </properties>
+ <dependencyManagement>
+ <dependencies>
+ </dependencies>
+ </dependencyManagement>
+ </profile>
+ <profile>
+ <id>hbase-hadoop1</id>
+ <activation>
+ <property>
+ <name>!hbase.profile</name>
+ </property>
+ </activation>
+ <properties>
+ <hbase.version>0.98.4-hadoop1</hbase.version>
+ </properties>
+ <dependencyManagement>
+ <dependencies>
+ </dependencies>
+ </dependencyManagement>
+ </profile>
+
</profiles>
<dependencies>
This caused the related compilation failed:
[INFO] Spark Project Examples ............................ FAILURE [0.102s]
RE: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by linkpatrickliu <li...@live.com>.
Hi,
You can set the settings in conf/spark-env.sh like this:export SPARK_LIBRARY_PATH=/usr/lib/hadoop/lib/native/
SPARK_JAVA_OPTS+="-Djava.library.path=$SPARK_LIBRARY_PATH "SPARK_JAVA_OPTS+="-Dspark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec "SPARK_JAVA_OPTS+="-Dio.compression.codecs=org.apache.hadoop.io.compress.SnappyCodec "
export SPARK_JAVA_OPTS
And make sure the snappy native library is located in "/usr/lib/hadoop/lib/native/" directory.
Also, if you are using 64 bit system, remember to re-compile the snappy native library to 64-bit, cause the official snappy lib is 32-bit.
$ file /usr/lib/hadoop/lib/native/*/usr/lib/hadoop/lib/native/libhadoop.a: current ar archive/usr/lib/hadoop/lib/native/libhadooppipes.a: current ar archive/usr/lib/hadoop/lib/native/libhadoop.so: symbolic link to `libhadoop.so.1.0.0'/usr/lib/hadoop/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped/usr/lib/hadoop/lib/native/libhadooputils.a: current ar archive/usr/lib/hadoop/lib/native/libhdfs.a: current ar archive/usr/lib/hadoop/lib/native/libsnappy.so: symbolic link to `libsnappy.so.1.1.3'/usr/lib/hadoop/lib/native/libsnappy.so.1: symbolic link to `libsnappy.so.1.1.3'/usr/lib/hadoop/lib/native/libsnappy.so.1.1.3: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
BRs,Patrick Liu
Date: Thu, 28 Aug 2014 18:36:25 -0700
From: ml-node+s1001560n13084h95@n3.nabble.com
To: linkpatrickliu@live.com
Subject: Re: org.apache.hadoop.io.compress.SnappyCodec not found
Hi, I fixed the issue by copying libsnappy.so to Java ire.
RegardsArthur
On 29 Aug, 2014, at 8:12 am, [hidden email] <[hidden email]> wrote:Hi,
If change my etc/hadoop/core-site.xml
from <property> <name>io.compression.codecs</name> <value> org.apache.hadoop.io.compress.SnappyCodec, org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec </value> </property>
to <property> <name>io.compression.codecs</name> <value> org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.SnappyCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec </value> </property>
and run the test again, I found this time it cannot find "org.apache.hadoop.io.compress.GzipCodec"
scala> inFILE.first()java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) at org.apache.spark.rdd.RDD.take(RDD.scala:983) at org.apache.spark.rdd.RDD.first(RDD.scala:1015) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15) at $iwC$$iwC$$iwC.<init>(<console>:20) at $iwC$$iwC.<init>(<console>:22) at $iwC.<init>(<console>:24) at <init>(<console>:26) at .<init>(<console>:30) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608) at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 55 moreCaused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.GzipCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) ... 60 moreCaused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.GzipCodec not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128) ... 62 more
Any idea to fix this issue?RegardsArthur
On 29 Aug, 2014, at 2:58 am, [hidden email] <[hidden email]> wrote:Hi,
my check native result:
hadoop checknative14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib libraryNative library checking:hadoop: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.sozlib: true /lib64/libz.so.1snappy: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1lz4: true revision:99bzip2: false
Any idea how to enable or disable snappy in Spark?
RegardsArthur
On 29 Aug, 2014, at 2:39 am, [hidden email] <[hidden email]> wrote:Hi,
I use Hadoop 2.4.1 and HBase 0.98.5 with snappy enabled in both Hadoop and HBase.With default setting in Spark 1.0.2, when trying to load a file I got "Class org.apache.hadoop.io.compress.SnappyCodec not found"
Can you please advise how to enable snappy in Spark?
RegardsArthur
scala> inFILE.first()java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) at org.apache.spark.rdd.RDD.take(RDD.scala:983) at org.apache.spark.rdd.RDD.first(RDD.scala:1015) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15) at $iwC$$iwC$$iwC.<init>(<console>:20) at $iwC$$iwC.<init>(<console>:22) at $iwC.<init>(<console>:24) at <init>(<console>:26) at .<init>(<console>:30) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608) at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 55 moreCaused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) ... 60 moreCaused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128) ... 62 more
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Compilaon-Error-Spark-1-0-2-with-HBase-0-98-tp12970p13084.html
To start a new topic under Apache Spark User List, email ml-node+s1001560n1h22@n3.nabble.com
To unsubscribe from Apache Spark User List, click here.
NAML
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Compilaon-Error-Spark-1-0-2-with-HBase-0-98-tp12970p13099.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I fixed the issue by copying libsnappy.so to Java ire.
Regards
Arthur
On 29 Aug, 2014, at 8:12 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> If change my etc/hadoop/core-site.xml
>
> from
> <property>
> <name>io.compression.codecs</name>
> <value>
> org.apache.hadoop.io.compress.SnappyCodec,
> org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> org.apache.hadoop.io.compress.BZip2Codec
> </value>
> </property>
>
> to
> <property>
> <name>io.compression.codecs</name>
> <value>
> org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.SnappyCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> org.apache.hadoop.io.compress.BZip2Codec
> </value>
> </property>
>
>
>
> and run the test again, I found this time it cannot find "org.apache.hadoop.io.compress.GzipCodec"
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.GzipCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.GzipCodec not found
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
> Any idea to fix this issue?
> Regards
> Arthur
>
>
> On 29 Aug, 2014, at 2:58 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> my check native result:
>>
>> hadoop checknative
>> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
>> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
>> Native library checking:
>> hadoop: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
>> zlib: true /lib64/libz.so.1
>> snappy: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
>> lz4: true revision:99
>> bzip2: false
>>
>> Any idea how to enable or disable snappy in Spark?
>>
>> Regards
>> Arthur
>>
>>
>> On 29 Aug, 2014, at 2:39 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I use Hadoop 2.4.1 and HBase 0.98.5 with snappy enabled in both Hadoop and HBase.
>>> With default setting in Spark 1.0.2, when trying to load a file I got "Class org.apache.hadoop.io.compress.SnappyCodec not found"
>>>
>>> Can you please advise how to enable snappy in Spark?
>>>
>>> Regards
>>> Arthur
>>>
>>>
>>> scala> inFILE.first()
>>> java.lang.RuntimeException: Error in configuring object
>>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>>> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>>> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>>> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
>>> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
>>> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
>>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
>>> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
>>> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
>>> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
>>> at $iwC$$iwC$$iwC.<init>(<console>:20)
>>> at $iwC$$iwC.<init>(<console>:22)
>>> at $iwC.<init>(<console>:24)
>>> at <init>(<console>:26)
>>> at .<init>(<console>:30)
>>> at .<clinit>(<console>)
>>> at .<init>(<console>:7)
>>> at .<clinit>(<console>)
>>> at $print(<console>)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
>>> at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
>>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
>>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
>>> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
>>> at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
>>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
>>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
>>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
>>> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
>>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
>>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
>>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
>>> at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
>>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
>>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>>> at org.apache.spark.repl.Main.main(Main.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
>>> ... 55 more
>>> Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
>>> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
>>> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
>>> at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
>>> ... 60 more
>>> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found
>>> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>>> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
>>> ... 62 more
>>>
>>>
>>
>
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
If change my etc/hadoop/core-site.xml
from
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.SnappyCodec,
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec
</value>
</property>
to
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.SnappyCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec
</value>
</property>
and run the test again, I found this time it cannot find "org.apache.hadoop.io.compress.GzipCodec"
scala> inFILE.first()
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.RDD.take(RDD.scala:983)
at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
at $iwC$$iwC$$iwC.<init>(<console>:20)
at $iwC$$iwC.<init>(<console>:22)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 55 more
Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.GzipCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 60 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.GzipCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
... 62 more
Any idea to fix this issue?
Regards
Arthur
On 29 Aug, 2014, at 2:58 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> my check native result:
>
> hadoop checknative
> 14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
> 14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
> Native library checking:
> hadoop: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
> zlib: true /lib64/libz.so.1
> snappy: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
>
> Any idea how to enable or disable snappy in Spark?
>
> Regards
> Arthur
>
>
> On 29 Aug, 2014, at 2:39 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
>
>> Hi,
>>
>> I use Hadoop 2.4.1 and HBase 0.98.5 with snappy enabled in both Hadoop and HBase.
>> With default setting in Spark 1.0.2, when trying to load a file I got "Class org.apache.hadoop.io.compress.SnappyCodec not found"
>>
>> Can you please advise how to enable snappy in Spark?
>>
>> Regards
>> Arthur
>>
>>
>> scala> inFILE.first()
>> java.lang.RuntimeException: Error in configuring object
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
>> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
>> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
>> at scala.Option.getOrElse(Option.scala:120)
>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
>> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
>> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
>> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
>> at $iwC$$iwC$$iwC.<init>(<console>:20)
>> at $iwC$$iwC.<init>(<console>:22)
>> at $iwC.<init>(<console>:24)
>> at <init>(<console>:26)
>> at .<init>(<console>:30)
>> at .<clinit>(<console>)
>> at .<init>(<console>:7)
>> at .<clinit>(<console>)
>> at $print(<console>)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
>> at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
>> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
>> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
>> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
>> at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
>> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
>> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
>> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
>> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
>> at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
>> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
>> at org.apache.spark.repl.Main$.main(Main.scala:31)
>> at org.apache.spark.repl.Main.main(Main.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
>> ... 55 more
>> Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
>> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
>> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
>> at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
>> ... 60 more
>> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found
>> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
>> ... 62 more
>>
>>
>
Re: org.apache.hadoop.io.compress.SnappyCodec not found
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
my check native result:
hadoop checknative
14/08/29 02:54:51 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
14/08/29 02:54:51 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libhadoop.so
zlib: true /lib64/libz.so.1
snappy: true /mnt/hadoop/hadoop-2.4.1_snappy/lib/native/Linux-amd64-64/libsnappy.so.1
lz4: true revision:99
bzip2: false
Any idea how to enable or disable snappy in Spark?
Regards
Arthur
On 29 Aug, 2014, at 2:39 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> I use Hadoop 2.4.1 and HBase 0.98.5 with snappy enabled in both Hadoop and HBase.
> With default setting in Spark 1.0.2, when trying to load a file I got "Class org.apache.hadoop.io.compress.SnappyCodec not found"
>
> Can you please advise how to enable snappy in Spark?
>
> Regards
> Arthur
>
>
> scala> inFILE.first()
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
> at org.apache.spark.rdd.RDD.take(RDD.scala:983)
> at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
> at $iwC$$iwC$$iwC.<init>(<console>:20)
> at $iwC$$iwC.<init>(<console>:22)
> at $iwC.<init>(<console>:24)
> at <init>(<console>:26)
> at .<init>(<console>:30)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
> at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 55 more
> Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
> at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
> ... 60 more
> Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
> ... 62 more
>
>
org.apache.hadoop.io.compress.SnappyCodec not found
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I use Hadoop 2.4.1 and HBase 0.98.5 with snappy enabled in both Hadoop and HBase.
With default setting in Spark 1.0.2, when trying to load a file I got "Class org.apache.hadoop.io.compress.SnappyCodec not found"
Can you please advise how to enable snappy in Spark?
Regards
Arthur
scala> inFILE.first()
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:158)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:171)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
at org.apache.spark.rdd.RDD.take(RDD.scala:983)
at org.apache.spark.rdd.RDD.first(RDD.scala:1015)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
at $iwC$$iwC$$iwC.<init>(<console>:20)
at $iwC$$iwC.<init>(<console>:22)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 55 more
Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.SnappyCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 60 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.SnappyCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
... 62 more
Re: SPARK-1297 patch error (spark-1297-v4.txt )
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi Ted,
I downloaded pom.xml to examples directory.
It works, thanks!!
Regards
Arthur
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [2.119s]
[INFO] Spark Project Core ................................ SUCCESS [1:27.100s]
[INFO] Spark Project Bagel ............................... SUCCESS [10.261s]
[INFO] Spark Project GraphX .............................. SUCCESS [31.332s]
[INFO] Spark Project ML Library .......................... SUCCESS [35.226s]
[INFO] Spark Project Streaming ........................... SUCCESS [39.135s]
[INFO] Spark Project Tools ............................... SUCCESS [6.469s]
[INFO] Spark Project Catalyst ............................ SUCCESS [36.521s]
[INFO] Spark Project SQL ................................. SUCCESS [35.488s]
[INFO] Spark Project Hive ................................ SUCCESS [35.296s]
[INFO] Spark Project REPL ................................ SUCCESS [18.668s]
[INFO] Spark Project YARN Parent POM ..................... SUCCESS [0.583s]
[INFO] Spark Project YARN Stable API ..................... SUCCESS [15.989s]
[INFO] Spark Project Assembly ............................ SUCCESS [11.497s]
[INFO] Spark Project External Twitter .................... SUCCESS [8.777s]
[INFO] Spark Project External Kafka ...................... SUCCESS [9.688s]
[INFO] Spark Project External Flume ...................... SUCCESS [10.411s]
[INFO] Spark Project External ZeroMQ ..................... SUCCESS [9.511s]
[INFO] Spark Project External MQTT ....................... SUCCESS [8.451s]
[INFO] Spark Project Examples ............................ SUCCESS [1:40.240s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:33.350s
[INFO] Finished at: Fri Aug 29 01:58:00 HKT 2014
[INFO] Final Memory: 82M/1086M
[INFO] ------------------------------------------------------------------------
On 29 Aug, 2014, at 1:36 am, Ted Yu <yu...@gmail.com> wrote:
> bq. Spark 1.0.2
>
> For the above release, you can download pom.xml attached to the JIRA and place it in examples directory
>
> I verified that the build against 0.98.4 worked using this command:
>
> mvn -Dhbase.profile=hadoop2 -Phadoop-2.4,yarn -Dhadoop.version=2.4.1 -DskipTests clean package
>
> Patch v5 is @ level 0 - you don't need to use -p1 in the patch command.
>
> Cheers
>
>
> On Thu, Aug 28, 2014 at 9:50 AM, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> I have just tried to apply the patch of SPARK-1297:
> https://issues.apache.org/jira/browse/SPARK-1297
>
> There are two files in it, named spark-1297-v2.txt and spark-1297-v4.txt respectively.
>
> When applying the 2nd one, I got "Hunk #1 FAILED at 45"
>
> Can you please advise how to fix it in order to make the compilation of Spark Project Examples success?
> (Here: Hadoop 2.4.1, HBase 0.98.5, Spark 1.0.2)
>
> Regards
> Arthur
>
>
>
> patch -p1 -i spark-1297-v4.txt
> patching file examples/pom.xml
> Hunk #1 FAILED at 45.
> Hunk #2 succeeded at 94 (offset -16 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
>
> below is the content of examples/pom.xml.rej:
> +++ examples/pom.xml
> @@ -45,6 +45,39 @@
> </dependency>
> </dependencies>
> </profile>
> + <profile>
> + <id>hbase-hadoop2</id>
> + <activation>
> + <property>
> + <name>hbase.profile</name>
> + <value>hadoop2</value>
> + </property>
> + </activation>
> + <properties>
> + <protobuf.version>2.5.0</protobuf.version>
> + <hbase.version>0.98.4-hadoop2</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> + <profile>
> + <id>hbase-hadoop1</id>
> + <activation>
> + <property>
> + <name>!hbase.profile</name>
> + </property>
> + </activation>
> + <properties>
> + <hbase.version>0.98.4-hadoop1</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> +
> </profiles>
>
> <dependencies>
>
>
> This caused the related compilation failed:
> [INFO] Spark Project Examples ............................ FAILURE [0.102s]
>
>
>
Re: SPARK-1297 patch error (spark-1297-v4.txt )
Posted by Ted Yu <yu...@gmail.com>.
bq. Spark 1.0.2
For the above release, you can download pom.xml attached to the JIRA and
place it in examples directory
I verified that the build against 0.98.4 worked using this command:
mvn -Dhbase.profile=hadoop2 -Phadoop-2.4,yarn -Dhadoop.version=2.4.1
-DskipTests clean package
Patch v5 is @ level 0 - you don't need to use -p1 in the patch command.
Cheers
On Thu, Aug 28, 2014 at 9:50 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have just tried to apply the patch of SPARK-1297:
> https://issues.apache.org/jira/browse/SPARK-1297
>
> There are two files in it, named spark-1297-v2.txt
> and spark-1297-v4.txt respectively.
>
> When applying the 2nd one, I got "Hunk #1 FAILED at 45"
>
> Can you please advise how to fix it in order to make the compilation
> of Spark Project Examples success?
> (Here: Hadoop 2.4.1, HBase 0.98.5, Spark 1.0.2)
>
> Regards
> Arthur
>
>
>
> patch -p1 -i spark-1297-v4.txt
> patching file examples/pom.xml
> Hunk #1 FAILED at 45.
> Hunk #2 succeeded at 94 (offset -16 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
>
> below is the content of examples/pom.xml.rej:
> +++ examples/pom.xml
> @@ -45,6 +45,39 @@
> </dependency>
> </dependencies>
> </profile>
> + <profile>
> + <id>hbase-hadoop2</id>
> + <activation>
> + <property>
> + <name>hbase.profile</name>
> + <value>hadoop2</value>
> + </property>
> + </activation>
> + <properties>
> + <protobuf.version>2.5.0</protobuf.version>
> + <hbase.version>0.98.4-hadoop2</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> + <profile>
> + <id>hbase-hadoop1</id>
> + <activation>
> + <property>
> + <name>!hbase.profile</name>
> + </property>
> + </activation>
> + <properties>
> + <hbase.version>0.98.4-hadoop1</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> +
> </profiles>
>
>
> <dependencies>
>
>
> This caused the related compilation failed:
> [INFO] Spark Project Examples ............................ FAILURE [0.102s]
>
>
>
Re: SPARK-1297 patch error (spark-1297-v4.txt )
Posted by Ted Yu <yu...@gmail.com>.
I attached patch v5 which corresponds to the pull request.
Please try again.
On Thu, Aug 28, 2014 at 9:50 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have just tried to apply the patch of SPARK-1297:
> https://issues.apache.org/jira/browse/SPARK-1297
>
> There are two files in it, named spark-1297-v2.txt
> and spark-1297-v4.txt respectively.
>
> When applying the 2nd one, I got "Hunk #1 FAILED at 45"
>
> Can you please advise how to fix it in order to make the compilation
> of Spark Project Examples success?
> (Here: Hadoop 2.4.1, HBase 0.98.5, Spark 1.0.2)
>
> Regards
> Arthur
>
>
>
> patch -p1 -i spark-1297-v4.txt
> patching file examples/pom.xml
> Hunk #1 FAILED at 45.
> Hunk #2 succeeded at 94 (offset -16 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
>
> below is the content of examples/pom.xml.rej:
> +++ examples/pom.xml
> @@ -45,6 +45,39 @@
> </dependency>
> </dependencies>
> </profile>
> + <profile>
> + <id>hbase-hadoop2</id>
> + <activation>
> + <property>
> + <name>hbase.profile</name>
> + <value>hadoop2</value>
> + </property>
> + </activation>
> + <properties>
> + <protobuf.version>2.5.0</protobuf.version>
> + <hbase.version>0.98.4-hadoop2</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> + <profile>
> + <id>hbase-hadoop1</id>
> + <activation>
> + <property>
> + <name>!hbase.profile</name>
> + </property>
> + </activation>
> + <properties>
> + <hbase.version>0.98.4-hadoop1</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> +
> </profiles>
>
>
> <dependencies>
>
>
> This caused the related compilation failed:
> [INFO] Spark Project Examples ............................ FAILURE [0.102s]
>
>
>
Re: SPARK-1297 patch error (spark-1297-v4.txt )
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
patch -p1 -i spark-1297-v5.txt
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git docs/building-with-maven.md docs/building-with-maven.md
|index 672d0ef..f8bcd2b 100644
|--- docs/building-with-maven.md
|+++ docs/building-with-maven.md
--------------------------
File to patch:
Please advise
Regards
Arthur
On 29 Aug, 2014, at 12:50 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> I have just tried to apply the patch of SPARK-1297:
> https://issues.apache.org/jira/browse/SPARK-1297
>
> There are two files in it, named spark-1297-v2.txt and spark-1297-v4.txt respectively.
>
> When applying the 2nd one, I got "Hunk #1 FAILED at 45"
>
> Can you please advise how to fix it in order to make the compilation of Spark Project Examples success?
> (Here: Hadoop 2.4.1, HBase 0.98.5, Spark 1.0.2)
>
> Regards
> Arthur
>
>
>
> patch -p1 -i spark-1297-v4.txt
> patching file examples/pom.xml
> Hunk #1 FAILED at 45.
> Hunk #2 succeeded at 94 (offset -16 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file examples/pom.xml.rej
>
> below is the content of examples/pom.xml.rej:
> +++ examples/pom.xml
> @@ -45,6 +45,39 @@
> </dependency>
> </dependencies>
> </profile>
> + <profile>
> + <id>hbase-hadoop2</id>
> + <activation>
> + <property>
> + <name>hbase.profile</name>
> + <value>hadoop2</value>
> + </property>
> + </activation>
> + <properties>
> + <protobuf.version>2.5.0</protobuf.version>
> + <hbase.version>0.98.4-hadoop2</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> + <profile>
> + <id>hbase-hadoop1</id>
> + <activation>
> + <property>
> + <name>!hbase.profile</name>
> + </property>
> + </activation>
> + <properties>
> + <hbase.version>0.98.4-hadoop1</hbase.version>
> + </properties>
> + <dependencyManagement>
> + <dependencies>
> + </dependencies>
> + </dependencyManagement>
> + </profile>
> +
> </profiles>
>
> <dependencies>
>
>
> This caused the related compilation failed:
> [INFO] Spark Project Examples ............................ FAILURE [0.102s]
>
>