spark failure

[Nathan Kronenfeld <nk...@oculusinfo.com>]

I'm using spark 0.8.1, and trying to run a job from a new remote client (it
works fine when run directly from the master).

When I try and run it, the job just fails without doing anything.

Unfortunately, I also can't find anywhere were it tells me why it fails.
 I'll add the bits of the logs below, but there really isn't much.

Does anyone know how to tell why it's failing? I assume it must be getting
an exception somewhere, but it isn't telling me about it.

On the client, I see:
14/02/24 23:44:43 INFO Client$ClientActor: Executor added:
app-20140224234441-0003/4 on
worker-20140224140443-hadoop-s2.oculus.local-40819
(hadoop-s2.oculus.local:7077) with 32 cores
14/02/24 23:44:43 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140224234441-0003/4 on hostPort hadoop-s2.oculus.local:7077 with 32
cores, 200.0 GB RAM
14/02/24 23:44:43 INFO Client$ClientActor: Executor updated:
app-20140224234441-0003/4 is now RUNNING
14/02/24 23:44:43 INFO FileInputFormat: Total input paths to process : 200
14/02/24 23:44:43 INFO Client$ClientActor: Executor updated:
app-20140224234441-0003/1 is now FAILED (Command exited with code 1)
14/02/24 23:44:43 INFO SparkDeploySchedulerBackend: Executor
app-20140224234441-0003/1 removed: Command exited with code 1

The master log just has:
14/02/24 23:44:43 INFO master.Master: Launching executor
app-20140224234441-0003/4 on worker
worker-20140224140443-hadoop-s2.oculus.local-40819
14/02/24 23:44:45 INFO master.Master: Removing executor
app-20140224234441-0003/4 because it is FAILED

(no other mention of 0003/4)

The client log has:
14/02/24 23:44:43 INFO worker.Worker: Asked to launch executor
app-20140224234441-0003/4 for Pyramid Binning(ndk)
14/02/24 23:44:43 INFO worker.ExecutorRunner: Launch command:
"/usr/java/jdk1.7.0_25-cloudera/bin/java" "-cp"
"math-utilities-0.2.jar:binning-utilities-0.2.jar:tile-generation-0.2.jar:hbase-client-0.95.2-cdh5.0.0-beta-1.jar:hbase-protocol-0.95.2-cdh5.0.0-beta-1.jar:hbase-common-0.95.2-cdh5.0.0-beta-1.jar:htrace-core-2.01.jar:avro-1.7.4.jar:commons-compress-1.4.1.jar:scala-library-2.9.3.jar:scala-compiler-2.9.3.jar:/opt/spark/conf:spark-assembly-0.8.1-incubating-hadoop2.2.0-mr1-cdh5.0.0-beta-1.jar"
"-Dspark.executor.memory=200G" "-Xms204800M" "-Xmx204800M"
"org.apache.spark.executor.CoarseGrainedExecutorBackend"
"akka://spark@hadoop-client.oculus.local:41101/user/CoarseGrainedScheduler"
"4" "hadoop-s2.oculus.local" "32" "app-20140224234441-0003"
14/02/24 23:44:45 INFO worker.Worker: Executor app-20140224234441-0003/4
finished with state FAILED message Command exited with code 1 exitStatus 1


Again, nothing else

-- 
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone:  +1-416-203-3003 x 238
Email:  nkronenfeld@oculusinfo.com