You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Raghuveer Chanda <ra...@gmail.com> on 2014/09/24 19:25:48 UTC
Spark with YARN
Hi,
I'm new to spark and facing problem with running a job in cluster using
YARN.
Initially i ran jobs using spark master as --master spark://dml2:7077 and
it is running fine on 3 workers.
But now im shifting to YARN, so installed YARN in cloud era on 3 node
cluster and changed the master to yarn-cluster but it is not working I
attached the screenshots of UI which are not progressing and just hanging
on.
Output on terminal :
This error is repeating
./spark-submit --class "class-name" --master yarn-cluster --num-executors 3
--executor-cores 3 jar-with-dependencies.jar
Do i need to configure YARN or why it is not getting all the workers ..
please help ...
14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
application identifier: application_1411578463780_0001
appId: 1
clientToAMToken: null
appDiagnostics:
appMasterHost: dml3
appQueue: root.chanda
appMasterRpcPort: 0
appStartTime: 1411578513545
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
appUser: chanda
14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
application identifier: application_1411578463780_0001
appId: 1
clientToAMToken: null
appDiagnostics:
appMasterHost: dml3
appQueue: root.chanda
appMasterRpcPort: 0
appStartTime: 1411578513545
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur
Re: Spark with YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
If you launched the job in yarn-cluster mode, the tracking URL is
printed on the output of the launched process. That will lead you to
the Spark UI once the job is running.
If you're using CM, you can reach the same link by clicking on the
"Resource Manager UI" link on your Yarn service, then finding the app
on the list and clicking on its "Application Master" link.
On Wed, Sep 24, 2014 at 12:07 PM, Raghuveer Chanda
<ra...@gmail.com> wrote:
> Yeah I got the logs and its reporting about the memory.
>
> 14/09/25 00:08:26 WARN YarnClusterScheduler: Initial job has not accepted
> any resources; check your cluster UI to ensure that workers are registered
> and have sufficient memory
>
> Now I shifted to big cluster with more memory but here im not able to view
> the UI while the job is running .. I need to check the status and spark UI
> of job.
>
> cse-hadoop-xx:18080 doesnt have running job it only had the jobs with master
> as spark://cse-hadoop-xx:7077 and not yarn-cluster .
>
> Im not ableto view the following links ..
> http://cse-hadoop-xx:50070/
> http://cse-hadoop-xx:8088/
>
> Is this due to some security option im not able to view the UI how can
> change it in cloudera.?
>
>
>
>
> On Thu, Sep 25, 2014 at 12:04 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>>
>> You need to use the command line yarn application that I mentioned
>> ("yarn logs"). You can't look at the logs through the UI after the app
>> stops.
>>
>> On Wed, Sep 24, 2014 at 11:16 AM, Raghuveer Chanda
>> <ra...@gmail.com> wrote:
>> >
>> > Thanks for the reply .. This is the error in the logs obtained from UI
>> > at
>> >
>> > http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_000001/chanda
>> >
>> > So now how to set the Log Server url ..
>> >
>> > Failed while trying to construct the redirect url to the log server. Log
>> > Server url may not be configured
>> >
>> > Container does not exist.
>> >
>> >
>> >
>> >
>> > On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin <va...@cloudera.com>
>> > wrote:
>> >>
>> >> You'll need to look at the driver output to have a better idea of
>> >> what's going on. You can use "yarn logs --applicationId blah" after
>> >> your app is finished (e.g. by killing it) to look at it.
>> >>
>> >> My guess is that your cluster doesn't have enough resources available
>> >> to service the container request you're making. That will show up in
>> >> the driver as periodic messages that no containers have been allocated
>> >> yet.
>> >>
>> >> On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
>> >> <ra...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I'm new to spark and facing problem with running a job in cluster
>> >> > using
>> >> > YARN.
>> >> >
>> >> > Initially i ran jobs using spark master as --master spark://dml2:7077
>> >> > and it
>> >> > is running fine on 3 workers.
>> >> >
>> >> > But now im shifting to YARN, so installed YARN in cloud era on 3 node
>> >> > cluster and changed the master to yarn-cluster but it is not working
>> >> > I
>> >> > attached the screenshots of UI which are not progressing and just
>> >> > hanging
>> >> > on.
>> >> >
>> >> > Output on terminal :
>> >> >
>> >> > This error is repeating
>> >> >
>> >> > ./spark-submit --class "class-name" --master yarn-cluster
>> >> > --num-executors 3
>> >> > --executor-cores 3 jar-with-dependencies.jar
>> >> >
>> >> >
>> >> > Do i need to configure YARN or why it is not getting all the workers
>> >> > ..
>> >> > please help ...
>> >> >
>> >> >
>> >> > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
>> >> > application identifier: application_1411578463780_0001
>> >> > appId: 1
>> >> > clientToAMToken: null
>> >> > appDiagnostics:
>> >> > appMasterHost: dml3
>> >> > appQueue: root.chanda
>> >> > appMasterRpcPort: 0
>> >> > appStartTime: 1411578513545
>> >> > yarnAppState: RUNNING
>> >> > distributedFinalState: UNDEFINED
>> >> > appTrackingUrl:
>> >> > http://dml2:8088/proxy/application_1411578463780_0001/
>> >> > appUser: chanda
>> >> > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
>> >> > application identifier: application_1411578463780_0001
>> >> > appId: 1
>> >> > clientToAMToken: null
>> >> > appDiagnostics:
>> >> > appMasterHost: dml3
>> >> > appQueue: root.chanda
>> >> > appMasterRpcPort: 0
>> >> > appStartTime: 1411578513545
>> >> > yarnAppState: RUNNING
>> >> > distributedFinalState: UNDEFINED
>> >> > appTrackingUrl:
>> >> > http://dml2:8088/proxy/application_1411578463780_0001/
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Raghuveer Chanda
>> >> > 4th year Undergraduate Student
>> >> > Computer Science and Engineering
>> >> > IIT Kharagpur
>> >> >
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> >> > For additional commands, e-mail: user-help@spark.apache.org
>> >>
>> >>
>> >>
>> >> --
>> >> Marcelo
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Raghuveer Chanda
>> > 4th year Undergraduate Student
>> > Computer Science and Engineering
>> > IIT Kharagpur
>>
>>
>>
>> --
>> Marcelo
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark with YARN
Posted by Raghuveer Chanda <ra...@gmail.com>.
Yeah I got the logs and its reporting about the memory.
14/09/25 00:08:26 WARN YarnClusterScheduler: Initial job has not accepted
any resources; check your cluster UI to ensure that workers are registered
and have sufficient memory
Now I shifted to big cluster with more memory but here im not able to view
the UI while the job is running .. I need to check the status and spark UI
of job.
cse-hadoop-xx:18080 doesnt have running job it only had the jobs with
master as spark://cse-hadoop-xx:7077 and not yarn-cluster .
Im not ableto view the following links ..
http://cse-hadoop-xx:50070/
http://cse-hadoop-xx:8088/
Is this due to some security option im not able to view the UI how can
change it in cloudera.?
On Thu, Sep 25, 2014 at 12:04 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> You need to use the command line yarn application that I mentioned
> ("yarn logs"). You can't look at the logs through the UI after the app
> stops.
>
> On Wed, Sep 24, 2014 at 11:16 AM, Raghuveer Chanda
> <ra...@gmail.com> wrote:
> >
> > Thanks for the reply .. This is the error in the logs obtained from UI at
> >
> http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_000001/chanda
> >
> > So now how to set the Log Server url ..
> >
> > Failed while trying to construct the redirect url to the log server. Log
> > Server url may not be configured
> >
> > Container does not exist.
> >
> >
> >
> >
> > On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin <va...@cloudera.com>
> > wrote:
> >>
> >> You'll need to look at the driver output to have a better idea of
> >> what's going on. You can use "yarn logs --applicationId blah" after
> >> your app is finished (e.g. by killing it) to look at it.
> >>
> >> My guess is that your cluster doesn't have enough resources available
> >> to service the container request you're making. That will show up in
> >> the driver as periodic messages that no containers have been allocated
> >> yet.
> >>
> >> On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
> >> <ra...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I'm new to spark and facing problem with running a job in cluster
> using
> >> > YARN.
> >> >
> >> > Initially i ran jobs using spark master as --master spark://dml2:7077
> >> > and it
> >> > is running fine on 3 workers.
> >> >
> >> > But now im shifting to YARN, so installed YARN in cloud era on 3 node
> >> > cluster and changed the master to yarn-cluster but it is not working I
> >> > attached the screenshots of UI which are not progressing and just
> >> > hanging
> >> > on.
> >> >
> >> > Output on terminal :
> >> >
> >> > This error is repeating
> >> >
> >> > ./spark-submit --class "class-name" --master yarn-cluster
> >> > --num-executors 3
> >> > --executor-cores 3 jar-with-dependencies.jar
> >> >
> >> >
> >> > Do i need to configure YARN or why it is not getting all the workers
> ..
> >> > please help ...
> >> >
> >> >
> >> > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> >> > application identifier: application_1411578463780_0001
> >> > appId: 1
> >> > clientToAMToken: null
> >> > appDiagnostics:
> >> > appMasterHost: dml3
> >> > appQueue: root.chanda
> >> > appMasterRpcPort: 0
> >> > appStartTime: 1411578513545
> >> > yarnAppState: RUNNING
> >> > distributedFinalState: UNDEFINED
> >> > appTrackingUrl:
> http://dml2:8088/proxy/application_1411578463780_0001/
> >> > appUser: chanda
> >> > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> >> > application identifier: application_1411578463780_0001
> >> > appId: 1
> >> > clientToAMToken: null
> >> > appDiagnostics:
> >> > appMasterHost: dml3
> >> > appQueue: root.chanda
> >> > appMasterRpcPort: 0
> >> > appStartTime: 1411578513545
> >> > yarnAppState: RUNNING
> >> > distributedFinalState: UNDEFINED
> >> > appTrackingUrl:
> http://dml2:8088/proxy/application_1411578463780_0001/
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Raghuveer Chanda
> >> > 4th year Undergraduate Student
> >> > Computer Science and Engineering
> >> > IIT Kharagpur
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> > For additional commands, e-mail: user-help@spark.apache.org
> >>
> >>
> >>
> >> --
> >> Marcelo
> >
> >
> >
> >
> > --
> > Regards,
> > Raghuveer Chanda
> > 4th year Undergraduate Student
> > Computer Science and Engineering
> > IIT Kharagpur
>
>
>
> --
> Marcelo
>
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur
Re: Spark with YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
You need to use the command line yarn application that I mentioned
("yarn logs"). You can't look at the logs through the UI after the app
stops.
On Wed, Sep 24, 2014 at 11:16 AM, Raghuveer Chanda
<ra...@gmail.com> wrote:
>
> Thanks for the reply .. This is the error in the logs obtained from UI at
> http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_000001/chanda
>
> So now how to set the Log Server url ..
>
> Failed while trying to construct the redirect url to the log server. Log
> Server url may not be configured
>
> Container does not exist.
>
>
>
>
> On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>>
>> You'll need to look at the driver output to have a better idea of
>> what's going on. You can use "yarn logs --applicationId blah" after
>> your app is finished (e.g. by killing it) to look at it.
>>
>> My guess is that your cluster doesn't have enough resources available
>> to service the container request you're making. That will show up in
>> the driver as periodic messages that no containers have been allocated
>> yet.
>>
>> On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
>> <ra...@gmail.com> wrote:
>> > Hi,
>> >
>> > I'm new to spark and facing problem with running a job in cluster using
>> > YARN.
>> >
>> > Initially i ran jobs using spark master as --master spark://dml2:7077
>> > and it
>> > is running fine on 3 workers.
>> >
>> > But now im shifting to YARN, so installed YARN in cloud era on 3 node
>> > cluster and changed the master to yarn-cluster but it is not working I
>> > attached the screenshots of UI which are not progressing and just
>> > hanging
>> > on.
>> >
>> > Output on terminal :
>> >
>> > This error is repeating
>> >
>> > ./spark-submit --class "class-name" --master yarn-cluster
>> > --num-executors 3
>> > --executor-cores 3 jar-with-dependencies.jar
>> >
>> >
>> > Do i need to configure YARN or why it is not getting all the workers ..
>> > please help ...
>> >
>> >
>> > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
>> > application identifier: application_1411578463780_0001
>> > appId: 1
>> > clientToAMToken: null
>> > appDiagnostics:
>> > appMasterHost: dml3
>> > appQueue: root.chanda
>> > appMasterRpcPort: 0
>> > appStartTime: 1411578513545
>> > yarnAppState: RUNNING
>> > distributedFinalState: UNDEFINED
>> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>> > appUser: chanda
>> > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
>> > application identifier: application_1411578463780_0001
>> > appId: 1
>> > clientToAMToken: null
>> > appDiagnostics:
>> > appMasterHost: dml3
>> > appQueue: root.chanda
>> > appMasterRpcPort: 0
>> > appStartTime: 1411578513545
>> > yarnAppState: RUNNING
>> > distributedFinalState: UNDEFINED
>> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Raghuveer Chanda
>> > 4th year Undergraduate Student
>> > Computer Science and Engineering
>> > IIT Kharagpur
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>> --
>> Marcelo
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark with YARN
Posted by Raghuveer Chanda <ra...@gmail.com>.
Thanks for the reply .. This is the error in the logs obtained from UI at
http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_000001/chanda
So now how to set the Log Server url ..
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configuredContainer does not exist.
On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> You'll need to look at the driver output to have a better idea of
> what's going on. You can use "yarn logs --applicationId blah" after
> your app is finished (e.g. by killing it) to look at it.
>
> My guess is that your cluster doesn't have enough resources available
> to service the container request you're making. That will show up in
> the driver as periodic messages that no containers have been allocated
> yet.
>
> On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
> <ra...@gmail.com> wrote:
> > Hi,
> >
> > I'm new to spark and facing problem with running a job in cluster using
> > YARN.
> >
> > Initially i ran jobs using spark master as --master spark://dml2:7077
> and it
> > is running fine on 3 workers.
> >
> > But now im shifting to YARN, so installed YARN in cloud era on 3 node
> > cluster and changed the master to yarn-cluster but it is not working I
> > attached the screenshots of UI which are not progressing and just hanging
> > on.
> >
> > Output on terminal :
> >
> > This error is repeating
> >
> > ./spark-submit --class "class-name" --master yarn-cluster
> --num-executors 3
> > --executor-cores 3 jar-with-dependencies.jar
> >
> >
> > Do i need to configure YARN or why it is not getting all the workers ..
> > please help ...
> >
> >
> > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> > application identifier: application_1411578463780_0001
> > appId: 1
> > clientToAMToken: null
> > appDiagnostics:
> > appMasterHost: dml3
> > appQueue: root.chanda
> > appMasterRpcPort: 0
> > appStartTime: 1411578513545
> > yarnAppState: RUNNING
> > distributedFinalState: UNDEFINED
> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> > appUser: chanda
> > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> > application identifier: application_1411578463780_0001
> > appId: 1
> > clientToAMToken: null
> > appDiagnostics:
> > appMasterHost: dml3
> > appQueue: root.chanda
> > appMasterRpcPort: 0
> > appStartTime: 1411578513545
> > yarnAppState: RUNNING
> > distributedFinalState: UNDEFINED
> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> >
> >
> >
> >
> > --
> > Regards,
> > Raghuveer Chanda
> > 4th year Undergraduate Student
> > Computer Science and Engineering
> > IIT Kharagpur
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
>
>
>
> --
> Marcelo
>
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur
Re: Spark with YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
You'll need to look at the driver output to have a better idea of
what's going on. You can use "yarn logs --applicationId blah" after
your app is finished (e.g. by killing it) to look at it.
My guess is that your cluster doesn't have enough resources available
to service the container request you're making. That will show up in
the driver as periodic messages that no containers have been allocated
yet.
On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
<ra...@gmail.com> wrote:
> Hi,
>
> I'm new to spark and facing problem with running a job in cluster using
> YARN.
>
> Initially i ran jobs using spark master as --master spark://dml2:7077 and it
> is running fine on 3 workers.
>
> But now im shifting to YARN, so installed YARN in cloud era on 3 node
> cluster and changed the master to yarn-cluster but it is not working I
> attached the screenshots of UI which are not progressing and just hanging
> on.
>
> Output on terminal :
>
> This error is repeating
>
> ./spark-submit --class "class-name" --master yarn-cluster --num-executors 3
> --executor-cores 3 jar-with-dependencies.jar
>
>
> Do i need to configure YARN or why it is not getting all the workers ..
> please help ...
>
>
> 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> appUser: chanda
> 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark with YARN
Posted by Raghuveer Chanda <ra...@gmail.com>.
The screenshot executors.8080.png is of the executors tab itself and only
driver is added without workers even if I kept the master as yarn-cluster.
On Wed, Sep 24, 2014 at 11:18 PM, Matt Narrell <ma...@gmail.com>
wrote:
> This just shows the driver. Click the Executors tab in the Spark UI
>
> mn
>
> On Sep 24, 2014, at 11:25 AM, Raghuveer Chanda <ra...@gmail.com>
> wrote:
>
> Hi,
>
> I'm new to spark and facing problem with running a job in cluster using
> YARN.
>
> Initially i ran jobs using spark master as --master spark://dml2:7077 and
> it is running fine on 3 workers.
>
> But now im shifting to YARN, so installed YARN in cloud era on 3 node
> cluster and changed the master to yarn-cluster but it is not working I
> attached the screenshots of UI which are not progressing and just hanging
> on.
>
> Output on terminal :
>
> This error is repeating
>
> ./spark-submit --class "class-name" --master yarn-cluster --num-executors
> 3 --executor-cores 3 jar-with-dependencies.jar
>
>
> Do i need to configure YARN or why it is not getting all the workers ..
> please help ...
>
>
> 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> appUser: chanda
> 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
> <executors.8088.PNG><spark.8088.PNG><yarn.8088.PNG>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur
Re: Spark with YARN
Posted by Matt Narrell <ma...@gmail.com>.
This just shows the driver. Click the Executors tab in the Spark UI
mn
On Sep 24, 2014, at 11:25 AM, Raghuveer Chanda <ra...@gmail.com> wrote:
> Hi,
>
> I'm new to spark and facing problem with running a job in cluster using YARN.
>
> Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers.
>
> But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on.
>
> Output on terminal :
>
> This error is repeating
>
> ./spark-submit --class "class-name" --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar
>
>
> Do i need to configure YARN or why it is not getting all the workers .. please help ...
>
>
> 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> appUser: chanda
> 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
> <executors.8088.PNG><spark.8088.PNG><yarn.8088.PNG>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
Re: Spark with YARN
Posted by Raghuveer Chanda <ra...@gmail.com>.
Thanks for the reply, I have doubt as to which path to set for
YARN_CONF_DIR
My /etc/hadoop folder has the following sub folders
conf conf.cloudera.hdfs conf.cloudera.mapreduce conf.cloudera.yarn
and both conf and conf.cloudera.yarn folders have yarn-site.xml. As of now
I set the variable as
export YARN_CONF_DIR=/etc/hadoop/conf
But no change in the error .. only driver is added no executors for it, do
i need to make any change in conf files or what is the correct path for
variable.
On Wed, Sep 24, 2014 at 11:18 PM, Greg Hill <gr...@rackspace.com> wrote:
> Do you have YARN_CONF_DIR set in your environment to point Spark to
> where your yarn configs are?
>
> Greg
>
> From: Raghuveer Chanda <ra...@gmail.com>
> Date: Wednesday, September 24, 2014 12:25 PM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Subject: Spark with YARN
>
> Hi,
>
> I'm new to spark and facing problem with running a job in cluster using
> YARN.
>
> Initially i ran jobs using spark master as --master spark://dml2:7077
> and it is running fine on 3 workers.
>
> But now im shifting to YARN, so installed YARN in cloud era on 3 node
> cluster and changed the master to yarn-cluster but it is not working I
> attached the screenshots of UI which are not progressing and just hanging
> on.
>
> Output on terminal :
>
> This error is repeating
>
> ./spark-submit --class "class-name" --master yarn-cluster
> --num-executors 3 --executor-cores 3 jar-with-dependencies.jar
>
>
> Do i need to configure YARN or why it is not getting all the workers ..
> please help ...
>
>
> 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
> appUser: chanda
> 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
> application identifier: application_1411578463780_0001
> appId: 1
> clientToAMToken: null
> appDiagnostics:
> appMasterHost: dml3
> appQueue: root.chanda
> appMasterRpcPort: 0
> appStartTime: 1411578513545
> yarnAppState: RUNNING
> distributedFinalState: UNDEFINED
> appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur
>
>
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur
Re: Spark with YARN
Posted by Greg Hill <gr...@RACKSPACE.COM>.
Do you have YARN_CONF_DIR set in your environment to point Spark to where your yarn configs are?
Greg
From: Raghuveer Chanda <ra...@gmail.com>>
Date: Wednesday, September 24, 2014 12:25 PM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: Spark with YARN
Hi,
I'm new to spark and facing problem with running a job in cluster using YARN.
Initially i ran jobs using spark master as --master spark://dml2:7077 and it is running fine on 3 workers.
But now im shifting to YARN, so installed YARN in cloud era on 3 node cluster and changed the master to yarn-cluster but it is not working I attached the screenshots of UI which are not progressing and just hanging on.
Output on terminal :
This error is repeating
./spark-submit --class "class-name" --master yarn-cluster --num-executors 3 --executor-cores 3 jar-with-dependencies.jar
Do i need to configure YARN or why it is not getting all the workers .. please help ...
14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
application identifier: application_1411578463780_0001
appId: 1
clientToAMToken: null
appDiagnostics:
appMasterHost: dml3
appQueue: root.chanda
appMasterRpcPort: 0
appStartTime: 1411578513545
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
appUser: chanda
14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
application identifier: application_1411578463780_0001
appId: 1
clientToAMToken: null
appDiagnostics:
appMasterHost: dml3
appQueue: root.chanda
appMasterRpcPort: 0
appStartTime: 1411578513545
yarnAppState: RUNNING
distributedFinalState: UNDEFINED
appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
--
Regards,
Raghuveer Chanda
4th year Undergraduate Student
Computer Science and Engineering
IIT Kharagpur