You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sjk <sh...@163.com> on 2016/03/14 14:55:20 UTC
mess spark cluster mode error
hi,all, when i run task on mesos, task error below. for help, thanks a lot.
cluster mode, command:
$SPARK_HOME/spark-submit --class com.xxx.ETL --master mesos://192.168.191.116:7077 --deploy-mode cluster --supervise --driver-memory 2G --executor-memory 10G —
total-executor-cores 4 http://jar.xxx.info/streaming-etl-assembly-1.0.jar
task stderr:
I0314 21:13:17.520845 29008 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/appweb","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/data\/program\/spark-1.6.0-bin-hadoop2.6.tgz"}}],"sandbox_directory":"\/data\/mesos\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/frameworks\/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044\/executors\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/runs\/92509aa4-7804-459b-857d-cfc08c31a993","user":"appweb"}
I0314 21:13:17.522541 29008 fetcher.cpp:379] Fetching URI '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
I0314 21:13:17.522562 29008 fetcher.cpp:250] Fetching directly into the sandbox directory
I0314 21:13:17.522586 29008 fetcher.cpp:187] Fetching URI '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
I0314 21:13:17.522603 29008 fetcher.cpp:167] Copying resource with command:cp '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
I0314 21:13:17.880008 29008 fetcher.cpp:84] Extracting with command: tar -C '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993' -xf '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
I0314 21:13:20.911213 29008 fetcher.cpp:92] Extracted '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz' into '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993'
I0314 21:13:20.911278 29008 fetcher.cpp:456] Fetched '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' to '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/launcher/Main
Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.launcher.Main. Program will exit.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: mess spark cluster mode error
Posted by sjk <sh...@163.com>.
when i change to default coarse-grained, it’s ok.
> On Mar 14, 2016, at 21:55, sjk <sh...@163.com> wrote:
>
> hi,all, when i run task on mesos, task error below. for help, thanks a lot.
>
>
> cluster mode, command:
>
> $SPARK_HOME/spark-submit --class com.xxx.ETL --master mesos://192.168.191.116:7077 --deploy-mode cluster --supervise --driver-memory 2G --executor-memory 10G —
> total-executor-cores 4 http://jar.xxx.info/streaming-etl-assembly-1.0.jar
>
>
> task stderr:
>
>
> I0314 21:13:17.520845 29008 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/appweb","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"\/data\/program\/spark-1.6.0-bin-hadoop2.6.tgz"}}],"sandbox_directory":"\/data\/mesos\/slaves\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/frameworks\/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044\/executors\/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7\/runs\/92509aa4-7804-459b-857d-cfc08c31a993","user":"appweb"}
> I0314 21:13:17.522541 29008 fetcher.cpp:379] Fetching URI '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.522562 29008 fetcher.cpp:250] Fetching directly into the sandbox directory
> I0314 21:13:17.522586 29008 fetcher.cpp:187] Fetching URI '/data/program/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.522603 29008 fetcher.cpp:167] Copying resource with command:cp '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:17.880008 29008 fetcher.cpp:84] Extracting with command: tar -C '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993' -xf '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> I0314 21:13:20.911213 29008 fetcher.cpp:92] Extracted '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz' into '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993'
> I0314 21:13:20.911278 29008 fetcher.cpp:456] Fetched '/data/program/spark-1.6.0-bin-hadoop2.6.tgz' to '/data/mesos/slaves/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/frameworks/dd8e95f7-3626-4e46-b48c-b3b58b573c4d-0044/executors/c2f100e1-13a8-40d9-a00f-68389300dfc1-S7/runs/92509aa4-7804-459b-857d-cfc08c31a993/spark-1.6.0-bin-hadoop2.6.tgz'
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/launcher/Main
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
> Could not find the main class: org.apache.spark.launcher.Main. Program will exit.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org