Update gcc version ,Still snappy error.

[buring <qy...@gmail.com>]

 I update the spark version form 1.02 to 1.10 , experienced an snappy version
issue with the new Spark-1.1.0. After update the glibc version, occured a
another issue. I abstract the log as follows:

14/09/25 11:29:18 WARN [org.apache.hadoop.util.NativeCodeLoader---main]:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
14/09/25 11:29:19 WARN [org.apache.hadoop.hdfs.DomainSocketFactory---main]:
The short-circuit local reads feature is disabled because libhadoop cannot
be loaded.

WARN [org.apache.spark.scheduler.TaskSetManager---Result resolver thread-0]:
Lost task 0.0 in stage 1.0 (TID 1, spark-dev134):
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
        org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:236)
        org.xerial.snappy.Snappy.<clinit>(Snappy.java:48)
       
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351)

WARN [org.apache.spark.scheduler.TaskSetManager---Result resolver thread-3]:
Lost task 4.0 in stage 1.0 (TID 4, spark-dev134):
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
       
org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351)
       
org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:159)
        org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)

14/09/25 11:29:24 ERROR
[org.apache.spark.network.ConnectionManager---handle-read-write-executor-3]:
Corresponding SendingConnection to ConnectionManagerId(spark-dev135,38649)
not found

14/09/25 11:29:24 INFO [org.apache.spark.scheduler.DAGScheduler---main]:
Failed to run count at SessionSVD2.scala:23
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure:
Lost task 2.3 in stage 1.0 (TID 9, spark-dev135): ExecutorLostFailure
(executor lost)
Driver stacktrace:
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

1)I tried to print JAVA_LIBRARY_PATH ,native-hadoop library is not in the 
path,and I set the
System.setProperty("JAVA_LIBRARY_PATH","hadoop_home/lib/native/"),which only
effect in System.getenv(),but not the
System.getProperty("JAVA_LIBRARY_PATH") .And "hadoop_home/lib/native/"
contain libhadoop libsnappy.so file,whcih I want to include in path.

2)I found in /tmp there are many snappy-uuuid file,each time i submit a job
it create a snappy-uuuid file. Before I update the glibc version,my fellow
update the snappy version,I think this is the reason why it can find the
snappy file but libhadoop.

Is there any ideas?
Thanks 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Update-gcc-version-Still-snappy-error-tp15137.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org