You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sy...@tsmc.com on 2016/06/24 11:01:36 UTC
problem running spark with yarn-client not using spark-submit
Hello guys,
Trying to run spark with yarn-client not using spark-submit here but the
jobs kept failed while AM launching executor.
The error collected by yarn like below.
Looks like some environment setting is missing?
Could someone help me out with this.
Thanks in advance!
HY Chung
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=256m; support was removed in 8.0
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass
(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass
(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass
(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main
(ApplicationMaster.scala:674)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main
(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass
(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
Best regards,
S.Y. Chung 鍾學毅
F14MITD
Taiwan Semiconductor Manufacturing Company, Ltd.
Tel: 06-5056688 Ext: 734-6325
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------
Re: problem running spark with yarn-client not using spark-submit
Posted by Saisai Shao <sa...@gmail.com>.
It means several jars are missing in the yarn container environment, if you
want to submit your application through some other ways besides
spark-submit, you have to take care all the environment things yourself.
Since we don't know your implementation of java web service, so it is hard
to provide useful information. Basically the problem is spark jar is
missing in the AM side, so you have to put spark jars into distributed
cache ahead of time.
On Mon, Jun 27, 2016 at 11:19 AM, <sy...@tsmc.com> wrote:
> Hi,
>
> Thanks for reply. It's a java web service resides in a jboss container.
>
> HY Chung
> Best regards,
>
> S.Y. Chung 鍾學毅
> F14MITD
> Taiwan Semiconductor Manufacturing Company, Ltd.
> Tel: 06-5056688 Ext: 734-6325
>
>
> |------------------------------------->
> | Mich Talebzadeh |
> | <mich.talebzadeh@gmail.co|
> | m> |
> | |
> | |
> | |
> | 2016/06/24 下午 08:05 |
> |------------------------------------->
>
> >-------------------------------------------------------------------------------------------------------------------------------------------------|
> |
> |
> |
> |
> |
> To|
> | sychungd@tsmc.com
> |
> |
> cc|
> | "user @spark" <us...@spark.apache.org>
> |
> |
> Subject|
> | [Spam][SMG] Re: problem running spark with yarn-client not
> using spark-submit
> |
> |
> |
> |
> |
> |
> |
> |
> |
> |
> |
>
> >-------------------------------------------------------------------------------------------------------------------------------------------------|
>
>
>
>
> Hi,
> Trying to run spark with yarn-client not using spark-submit here
>
> what are you using to submit the job? spark-shell, spark-sql or anything
> else
>
>
> Dr Mich Talebzadeh
>
> LinkedIn
>
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>
>
>
>
> On 24 June 2016 at 12:01, <sy...@tsmc.com> wrote:
>
> Hello guys,
>
> Trying to run spark with yarn-client not using spark-submit here but the
> jobs kept failed while AM launching executor.
> The error collected by yarn like below.
> Looks like some environment setting is missing?
> Could someone help me out with this.
>
> Thanks in advance!
> HY Chung
>
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256m; support was removed in 8.0
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/Logging
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass
> (ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass
> (SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass
> (URLClassLoader.java:455)
> at java.net.URLClassLoader.access$100
> (URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run
> (URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run
> (URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at java.net.URLClassLoader.findClass
> (URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass
> (Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.spark.deploy.yarn.ExecutorLauncher$.main
> (ApplicationMaster.scala:674)
> at org.apache.spark.deploy.yarn.ExecutorLauncher.main
> (ApplicationMaster.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
> at java.net.URLClassLoader$1.run
> (URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run
> (URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at java.net.URLClassLoader.findClass
> (URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass
> (Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 14 more
>
>
> Best regards,
>
> S.Y. Chung 鍾學毅
> F14MITD
> Taiwan Semiconductor Manufacturing Company, Ltd.
> Tel: 06-5056688 Ext: 734-6325
>
>
> ---------------------------------------------------------------------------
>
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary
> information
> for the sole use of its
> intended recipient. Any unauthorized review, use or distribution by
> anyone
> other than the intended
> recipient is strictly prohibited. If you are not the intended
> recipient,
> please notify the sender by
> replying to this email, and then delete this email and any copies of it
> immediately. Thank you.
>
>
> ---------------------------------------------------------------------------
>
>
>
>
>
>
> ---------------------------------------------------------------------------
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary information
> for the sole use of its
> intended recipient. Any unauthorized review, use or distribution by anyone
> other than the intended
> recipient is strictly prohibited. If you are not the intended recipient,
> please notify the sender by
> replying to this email, and then delete this email and any copies of it
> immediately. Thank you.
>
> ---------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: problem running spark with yarn-client not using spark-submit
Posted by sy...@tsmc.com.
Hi,
Thanks for reply. It's a java web service resides in a jboss container.
HY Chung
Best regards,
S.Y. Chung 鍾學毅
F14MITD
Taiwan Semiconductor Manufacturing Company, Ltd.
Tel: 06-5056688 Ext: 734-6325
|------------------------------------->
| Mich Talebzadeh |
| <mich.talebzadeh@gmail.co|
| m> |
| |
| |
| |
| 2016/06/24 下午 08:05 |
|------------------------------------->
>-------------------------------------------------------------------------------------------------------------------------------------------------|
| |
| |
| To|
| sychungd@tsmc.com |
| cc|
| "user @spark" <us...@spark.apache.org> |
| Subject|
| [Spam][SMG] Re: problem running spark with yarn-client not using spark-submit |
| |
| |
| |
| |
| |
>-------------------------------------------------------------------------------------------------------------------------------------------------|
Hi,
Trying to run spark with yarn-client not using spark-submit here
what are you using to submit the job? spark-shell, spark-sql or anything
else
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
On 24 June 2016 at 12:01, <sy...@tsmc.com> wrote:
Hello guys,
Trying to run spark with yarn-client not using spark-submit here but the
jobs kept failed while AM launching executor.
The error collected by yarn like below.
Looks like some environment setting is missing?
Could someone help me out with this.
Thanks in advance!
HY Chung
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=256m; support was removed in 8.0
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass
(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass
(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass
(URLClassLoader.java:455)
at java.net.URLClassLoader.access$100
(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run
(URLClassLoader.java:367)
at java.net.URLClassLoader$1.run
(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native
Method)
at java.net.URLClassLoader.findClass
(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass
(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main
(ApplicationMaster.scala:674)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main
(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader$1.run
(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run
(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native
Method)
at java.net.URLClassLoader.findClass
(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass
(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
Best regards,
S.Y. Chung 鍾學毅
F14MITD
Taiwan Semiconductor Manufacturing Company, Ltd.
Tel: 06-5056688 Ext: 734-6325
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary
information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by
anyone
other than the intended
recipient is strictly prohibited. If you are not the intended
recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: problem running spark with yarn-client not using spark-submit
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi,
Trying to run spark with yarn-client not using spark-submit here
what are you using to submit the job? spark-shell, spark-sql or anything
else
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 24 June 2016 at 12:01, <sy...@tsmc.com> wrote:
>
> Hello guys,
>
> Trying to run spark with yarn-client not using spark-submit here but the
> jobs kept failed while AM launching executor.
> The error collected by yarn like below.
> Looks like some environment setting is missing?
> Could someone help me out with this.
>
> Thanks in advance!
> HY Chung
>
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256m; support was removed in 8.0
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/Logging
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> at java.security.SecureClassLoader.defineClass
> (SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass
> (URLClassLoader.java:455)
> at
> java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:367)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass
> (Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at org.apache.spark.deploy.yarn.ExecutorLauncher$.main
> (ApplicationMaster.scala:674)
> at org.apache.spark.deploy.yarn.ExecutorLauncher.main
> (ApplicationMaster.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass
> (Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 14 more
>
>
> Best regards,
>
> S.Y. Chung 鍾學毅
> F14MITD
> Taiwan Semiconductor Manufacturing Company, Ltd.
> Tel: 06-5056688 Ext: 734-6325
>
> ---------------------------------------------------------------------------
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary information
> for the sole use of its
> intended recipient. Any unauthorized review, use or distribution by anyone
> other than the intended
> recipient is strictly prohibited. If you are not the intended recipient,
> please notify the sender by
> replying to this email, and then delete this email and any copies of it
> immediately. Thank you.
>
> ---------------------------------------------------------------------------
>
>
>