You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sachin Singh <sa...@gmail.com> on 2015/02/18 11:39:44 UTC
issue Running Spark Job on Yarn Cluster
Hi,
I want to run my spark Job in Hadoop yarn Cluster mode,
I am using below command -
spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g
--executor-cores 1 --class com.dc.analysis.jobs.AggregationJob
sparkanalitic.jar param1 param2 param3
I am getting error as under, kindly suggest whats going wrong ,is command is
proper or not ,thanks in advance,
Exception in thread "main" org.apache.spark.SparkException: Application
finished with failed status
at
org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:509)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: issue Running Spark Job on Yarn Cluster
Posted by Sachin Singh <sa...@gmail.com>.
Yes.
On 19 Feb 2015 23:40, "Harshvardhan Chauhan" <ha...@gumgum.com> wrote:
> Is this the full stack trace ?
>
> On Wed, Feb 18, 2015 at 2:39 AM, sachin Singh <sa...@gmail.com>
> wrote:
>
>> Hi,
>> I want to run my spark Job in Hadoop yarn Cluster mode,
>> I am using below command -
>> spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g
>> --executor-cores 1 --class com.dc.analysis.jobs.AggregationJob
>> sparkanalitic.jar param1 param2 param3
>> I am getting error as under, kindly suggest whats going wrong ,is command
>> is
>> proper or not ,thanks in advance,
>>
>> Exception in thread "main" org.apache.spark.SparkException: Application
>> finished with failed status
>> at
>> org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:509)
>> at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
>> at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
>> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
> *Harshvardhan Chauhan* | Software Engineer
> *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> 310-260-9666 | harsh@gumgum.com
>
Re: issue Running Spark Job on Yarn Cluster
Posted by Harshvardhan Chauhan <ha...@gumgum.com>.
Is this the full stack trace ?
On Wed, Feb 18, 2015 at 2:39 AM, sachin Singh <sa...@gmail.com>
wrote:
> Hi,
> I want to run my spark Job in Hadoop yarn Cluster mode,
> I am using below command -
> spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g
> --executor-cores 1 --class com.dc.analysis.jobs.AggregationJob
> sparkanalitic.jar param1 param2 param3
> I am getting error as under, kindly suggest whats going wrong ,is command
> is
> proper or not ,thanks in advance,
>
> Exception in thread "main" org.apache.spark.SparkException: Application
> finished with failed status
> at
> org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:509)
> at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
> at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
*Harshvardhan Chauhan* | Software Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com
Re: issue Running Spark Job on Yarn Cluster
Posted by Marcelo Vanzin <va...@cloudera.com>.
You'll need to look at your application's logs. You can use "yarn logs
--applicationId [id]" to see them.
On Wed, Feb 18, 2015 at 2:39 AM, sachin Singh <sa...@gmail.com> wrote:
> Hi,
> I want to run my spark Job in Hadoop yarn Cluster mode,
> I am using below command -
> spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g
> --executor-cores 1 --class com.dc.analysis.jobs.AggregationJob
> sparkanalitic.jar param1 param2 param3
> I am getting error as under, kindly suggest whats going wrong ,is command is
> proper or not ,thanks in advance,
>
> Exception in thread "main" org.apache.spark.SparkException: Application
> finished with failed status
> at
> org.apache.spark.deploy.yarn.ClientBase$class.run(ClientBase.scala:509)
> at org.apache.spark.deploy.yarn.Client.run(Client.scala:35)
> at org.apache.spark.deploy.yarn.Client$.main(Client.scala:139)
> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: issue Running Spark Job on Yarn Cluster
Posted by roni <ro...@gmail.com>.
look at the logs
yarn logs --applicationId <applicationId>
That should give the error.
On Wed, Mar 4, 2015 at 9:21 AM, sachin Singh <sa...@gmail.com>
wrote:
> Not yet,
> Please let. Me know if you found solution,
>
> Regards
> Sachin
> On 4 Mar 2015 21:45, "mael2210 [via Apache Spark User List]" <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=21912&i=0>> wrote:
>
>> Hello,
>>
>> I am facing the exact same issue. Could you solve the problem ?
>>
>> Kind regards
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697p21909.html
>> To unsubscribe from issue Running Spark Job on Yarn Cluster, click here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
> ------------------------------
> View this message in context: Re: issue Running Spark Job on Yarn Cluster
> <http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697p21912.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
Re: issue Running Spark Job on Yarn Cluster
Posted by sachin Singh <sa...@gmail.com>.
Not yet,
Please let. Me know if you found solution,
Regards
Sachin
On 4 Mar 2015 21:45, "mael2210 [via Apache Spark User List]" <
ml-node+s1001560n21909h29@n3.nabble.com> wrote:
> Hello,
>
> I am facing the exact same issue. Could you solve the problem ?
>
> Kind regards
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697p21909.html
> To unsubscribe from issue Running Spark Job on Yarn Cluster, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=21697&code=c2FjaGluLnNoYXNoaUBnbWFpbC5jb218MjE2OTd8MTkyMzgyNjU3Mw==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/issue-Running-Spark-Job-on-Yarn-Cluster-tp21697p21912.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.