You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sarlindo <sa...@hotmail.com> on 2016/09/06 15:15:10 UTC

spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

I have 2 questions/issues.

1. We had the spark-master shut down (reason unknown) we looked at the
spark-master logs and it simply shows this, is there some other log I should
be looking at to find out why the master went down?

16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181. Will not attempt to
authenticate using SASL (unknown error)
16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
shutting down.
16/09/05 21:10:00 INFO ClientCnxn: Socket connection established, initiating
session, client: /142.201.219.75:56361, server:
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181


2. Spark 1.6.0 web console shows a running application in a "waiting"
status, but it's actually running. Is this an existing bug?

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png> 





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by arlindo santos <sa...@hotmail.com>.
Assuming you configured spark to use zookeeper for ha, when the master fails over to another node, the workers will automatically attach themselves to the newly elected master and this works fine. My issue is that once I go over to the new master web GUI ( I see all the workers attached just fine, this means the failover worked just fine) My issue is that the web GUI now think the spark streaming app running on the cluster is in "waiting" state and this is not the case because the app is actually running and processing events.


[http://ttllxapp-spk03.lab.tsx.com:18080/static/spark-logo-77x50px-hd.png] 1.6.0 <http://ttllxapp-spk03.lab.tsx.com:18080/> Spark Master at spark://10.142.191.154:7077

  *   URL: spark://10.142.191.154:7077
  *   REST URL: spark://10.142.191.154:6066 (cluster mode)
  *   Alive Workers: 3
  *   Cores in use: 12 Total, 12 Used
  *   Memory in use: 8.2 GB Total, 4.0 GB Used
  *   Applications: 1 Running, 0 Completed
  *   Drivers: 1 Running, 0 Completed
  *   Status: ALIVE

Workers
Worker Id       Address State   Cores   Memory
worker-20160907122724-10.142.191.154-7078<http://10.142.191.154:18081/> 10.142.191.154:7078     ALIVE   4 (4 Used)      2.7 GB (2.0 GB Used)
worker-20160907122724-10.142.191.159-7078<http://10.142.191.159:18081/> 10.142.191.159:7078     ALIVE   4 (4 Used)      2.7 GB (1024.0 MB Used)
worker-20160907122724-10.142.191.162-7078<http://10.142.191.162:18081/> 10.142.191.162:7078     ALIVE   4 (4 Used)      2.7 GB (1024.0 MB Used)
Running Applications
Application ID  Name    Cores   Memory per Node Submitted Time  User    State   Duration
app-20160907122851-0000<http://ttllxapp-spk03.lab.tsx.com:18080/app?appId=app-20160907122851-0000>
(kill)<http://ttllxapp-spk03.lab.tsx.com:18080/#>
        Ex1Feed<http://10.142.191.154:4040/>    12      1024.0 MB       2016/09/07 12:28:51     spark   WAITING 10 min




________________________________
From: Mich Talebzadeh <mi...@gmail.com>
Sent: September 7, 2016 3:42 PM
To: arlindo santos
Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

I just tested it.

If you start master on the original host, the workers on that host they won't respond. They will stay stale. So there is no heartbeat between workers and master except the initial handshake

The only way is to stop workers (if they are still running), restart the master and restart workers.


HTH



Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 7 September 2016 at 16:15, Mich Talebzadeh <mi...@gmail.com>> wrote:
Ok but look at the worker why is it still saying port 7077. That port on that host as far as I know is the port that local master is running which is no longer there?


Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 7 September 2016 at 16:05, arlindo santos <sa...@hotmail.com>> wrote:

Port 7077 is for "client" mode connections to the master. In "cluster" mode it's 6066 and this means the "driver" runs on the spark cluster on a node spark chooses. The command I use to deploy my spark app (including the driver) is below:


spark-submit --deploy-mode cluster --master spark://tiplxapp-spk01:6066,tiplxapp-spk02:6066,tiplxapp-spk03:6066 /app/tmx/ngxspark/lib/EX1AppSpark-1.0.13.jar



Yes, your right I believe when the master dies, zookeeper detects that and elects a new master node and spark-submit should carry on. Not sure how this leads into the UI believing the app is in "waiting" state?


Also, I noticed when these fail overs happen the "worker" web GUI goes a bit strange and starts reporting over allocated resources? Look at the cores and memory used?


[http://142.201.185.134:18081/static/spark-logo-77x50px-hd.png] 1.6.0 <http://142.201.185.134:18081/> Spark Worker at 142.201.185.134:7078<http://142.201.185.134:7078>

  *   ID: worker-20160622152457-142.201.185.134-7078
  *   Master URL: spark://142.201.185.132:7077<http://142.201.185.132:7077>
  *   Cores: 4 (5 Used)
  *   Memory: 2.7 GB (3.0 GB Used)

Back to Master<http://142.201.185.132:18080/>





________________________________
From: Mich Talebzadeh <mi...@gmail.com>>
Sent: September 7, 2016 2:52 PM
To: arlindo santos

Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

This is my take.

When you issue spark-submit on any node it start GUI on port 4040 by default. Otherwise you can specify port yourself with --conf  "spark.ui.port=<port>"

As I understand in standalone mode executors run on workers.

$SPARK_HOME/sbin/start-slave.sh spark://<host>::7077

That port 7077 is the master port. If master dies, then those workers lose connection to port 7077 so I believe they go stale. So the spark-submit carries on using the remaining executors on other workers.

So in summary one expects the job to run.  You start your UI on <HOST>:port.

One test you can do is to exit from UI and start UI on the host that zookeeper selects the master on the same port. That should work.

HTH







Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 7 September 2016 at 15:27, arlindo santos <sa...@hotmail.com>> wrote:
Yes refreshed a few times. Running in cluster mode.

Fyi.. I can duplicate this easily now. Our setup consists of 3 nodes running standalone spark, master and worker on each, zookeeper doing master leader election. If I kill a master on any node, the master shifts to another node and that is when the app state changes to waiting and never changes back to running on the gui, but really it's in a running mode.

Sent from my BlackBerry 10 smartphone on the Rogers network.
From: Mich Talebzadeh
Sent: Wednesday, September 7, 2016 9:50 AM
To: sarlindo
Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running


Have you refreshed the Spark UI page?

What Mode are you running your Spark app?

HTH


Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com>> wrote:
I have 2 questions/issues.

1. We had the spark-master shut down (reason unknown) we looked at the
spark-master logs and it simply shows this, is there some other log I should
be looking at to find out why the master went down?

16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>. Will not attempt to
authenticate using SASL (unknown error)
16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
shutting down.
16/09/05 21:10:00 INFO ClientCnxn: Socket connection established, initiating
session, client: /142.201.219.75:56361<http://142.201.219.75:56361>, server:
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>


2. Spark 1.6.0 web console shows a running application in a "waiting"
status, but it's actually running. Is this an existing bug?

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png>





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>






Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by Mich Talebzadeh <mi...@gmail.com>.
I just tested it.

If you start master on the original host, the workers on that host they
won't respond. They will stay stale. So there is no heartbeat between
workers and master except the initial handshake

The only way is to stop workers (if they are still running), restart the
master and restart workers.


HTH


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 7 September 2016 at 16:15, Mich Talebzadeh <mi...@gmail.com>
wrote:

> Ok but look at the worker why is it still saying port 7077. That port on
> that host as far as I know is the port that local master is running which
> is no longer there?
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 7 September 2016 at 16:05, arlindo santos <sa...@hotmail.com> wrote:
>
>> Port 7077 is for "client" mode connections to the master. In "cluster"
>> mode it's 6066 and this means the "driver" runs on the spark cluster on a
>> node spark chooses. The command I use to deploy my spark app (including the
>> driver) is below:
>>
>>
>> spark-submit --deploy-mode cluster --master spark://tiplxapp-spk01:6066,ti
>> plxapp-spk02:6066,tiplxapp-spk03:6066 /app/tmx/ngxspark/lib/EX1AppSp
>> ark-1.0.13.jar
>>
>>
>>
>> Yes, your right I believe when the master dies, zookeeper detects that
>> and elects a new master node and spark-submit should carry on. Not sure how
>> this leads into the UI believing the app is in "waiting" state?
>>
>>
>> Also, I noticed when these fail overs happen the "worker" web GUI goes a
>> bit strange and starts reporting over allocated resources? Look at the
>> cores and memory used?
>>
>>
>>  1.6.0  <http://142.201.185.134:18081/>Spark Worker at
>> 142.201.185.134:7078
>>
>>    - *ID:* worker-20160622152457-142.201.185.134-7078
>>    - *Master URL:* spark://142.201.185.132:7077
>>    - *Cores:* 4 (5 Used)
>>    - *Memory:* 2.7 GB (3.0 GB Used)
>>
>> Back to Master <http://142.201.185.132:18080/>
>>
>>
>>
>>
>>
>> ------------------------------
>> *From:* Mich Talebzadeh <mi...@gmail.com>
>> *Sent:* September 7, 2016 2:52 PM
>> *To:* arlindo santos
>>
>> *Cc:* user @spark
>> *Subject:* Re: spark 1.6.0 web console shows running application in a
>> "waiting" status, but it's acutally running
>>
>> This is my take.
>>
>> When you issue spark-submit on any node it start GUI on port 4040 by
>> default. Otherwise you can specify port yourself with --conf
>> "spark.ui.port=<port>"
>>
>> As I understand in standalone mode executors run on workers.
>>
>> $SPARK_HOME/sbin/start-slave.sh spark://<host>::7077
>>
>> That port 7077 is the master port. If master dies, then those workers
>> lose connection to port 7077 so I believe they go stale. So the
>> spark-submit carries on using the remaining executors on other workers.
>>
>> So in summary one expects the job to run.  You start your UI on
>> <HOST>:port.
>>
>> One test you can do is to exit from UI and start UI on the host that
>> zookeeper selects the master on the same port. That should work.
>>
>> HTH
>>
>>
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 7 September 2016 at 15:27, arlindo santos <sa...@hotmail.com>
>> wrote:
>>
>>> Yes refreshed a few times. Running in cluster mode.
>>>
>>> Fyi.. I can duplicate this easily now. Our setup consists of 3 nodes
>>> running standalone spark, master and worker on each, zookeeper doing master
>>> leader election. If I kill a master on any node, the master shifts to
>>> another node and that is when the app state changes to waiting and never
>>> changes back to running on the gui, but really it's in a running mode.
>>>
>>> Sent from my BlackBerry 10 smartphone on the Rogers network.
>>> *From: *Mich Talebzadeh
>>> *Sent: *Wednesday, September 7, 2016 9:50 AM
>>> *To: *sarlindo
>>> *Cc: *user @spark
>>> *Subject: *Re: spark 1.6.0 web console shows running application in a
>>> "waiting" status, but it's acutally running
>>>
>>> Have you refreshed the Spark UI page?
>>>
>>> What Mode are you running your Spark app?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com> wrote:
>>>
>>>> I have 2 questions/issues.
>>>>
>>>> 1. We had the spark-master shut down (reason unknown) we looked at the
>>>> spark-master logs and it simply shows this, is there some other log I
>>>> should
>>>> be looking at to find out why the master went down?
>>>>
>>>> 16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
>>>> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181. Will not attempt to
>>>> authenticate using SASL (unknown error)
>>>> 16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
>>>> shutting down.
>>>> 16/09/05 21:10:00 INFO ClientCnxn: Socket connection established,
>>>> initiating
>>>> session, client: /142.201.219.75:56361, server:
>>>> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181
>>>>
>>>>
>>>> 2. Spark 1.6.0 web console shows a running application in a "waiting"
>>>> status, but it's actually running. Is this an existing bug?
>>>>
>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.
>>>> 1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-
>>>> application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
>>
>

Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by arlindo santos <sa...@hotmail.com>.
Port 7077 is for "client" mode connections to the master. In "cluster" mode it's 6066 and this means the "driver" runs on the spark cluster on a node spark chooses. The command I use to deploy my spark app (including the driver) is below:


spark-submit --deploy-mode cluster --master spark://tiplxapp-spk01:6066,tiplxapp-spk02:6066,tiplxapp-spk03:6066 /app/tmx/ngxspark/lib/EX1AppSpark-1.0.13.jar



Yes, your right I believe when the master dies, zookeeper detects that and elects a new master node and spark-submit should carry on. Not sure how this leads into the UI believing the app is in "waiting" state?


Also, I noticed when these fail overs happen the "worker" web GUI goes a bit strange and starts reporting over allocated resources? Look at the cores and memory used?


[http://142.201.185.134:18081/static/spark-logo-77x50px-hd.png] 1.6.0 <http://142.201.185.134:18081/> Spark Worker at 142.201.185.134:7078

  *   ID: worker-20160622152457-142.201.185.134-7078
  *   Master URL: spark://142.201.185.132:7077
  *   Cores: 4 (5 Used)
  *   Memory: 2.7 GB (3.0 GB Used)

Back to Master<http://142.201.185.132:18080/>





________________________________
From: Mich Talebzadeh <mi...@gmail.com>
Sent: September 7, 2016 2:52 PM
To: arlindo santos
Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

This is my take.

When you issue spark-submit on any node it start GUI on port 4040 by default. Otherwise you can specify port yourself with --conf  "spark.ui.port=<port>"

As I understand in standalone mode executors run on workers.

$SPARK_HOME/sbin/start-slave.sh spark://<host>::7077

That port 7077 is the master port. If master dies, then those workers lose connection to port 7077 so I believe they go stale. So the spark-submit carries on using the remaining executors on other workers.

So in summary one expects the job to run.  You start your UI on <HOST>:port.

One test you can do is to exit from UI and start UI on the host that zookeeper selects the master on the same port. That should work.

HTH







Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 7 September 2016 at 15:27, arlindo santos <sa...@hotmail.com>> wrote:
Yes refreshed a few times. Running in cluster mode.

Fyi.. I can duplicate this easily now. Our setup consists of 3 nodes running standalone spark, master and worker on each, zookeeper doing master leader election. If I kill a master on any node, the master shifts to another node and that is when the app state changes to waiting and never changes back to running on the gui, but really it's in a running mode.

Sent from my BlackBerry 10 smartphone on the Rogers network.
From: Mich Talebzadeh
Sent: Wednesday, September 7, 2016 9:50 AM
To: sarlindo
Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running


Have you refreshed the Spark UI page?

What Mode are you running your Spark app?

HTH


Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com>> wrote:
I have 2 questions/issues.

1. We had the spark-master shut down (reason unknown) we looked at the
spark-master logs and it simply shows this, is there some other log I should
be looking at to find out why the master went down?

16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>. Will not attempt to
authenticate using SASL (unknown error)
16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
shutting down.
16/09/05 21:10:00 INFO ClientCnxn: Socket connection established, initiating
session, client: /142.201.219.75:56361<http://142.201.219.75:56361>, server:
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>


2. Spark 1.6.0 web console shows a running application in a "waiting"
status, but it's actually running. Is this an existing bug?

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png>





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>




Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by Mich Talebzadeh <mi...@gmail.com>.
This is my take.

When you issue spark-submit on any node it start GUI on port 4040 by
default. Otherwise you can specify port yourself with --conf
"spark.ui.port=<port>"

As I understand in standalone mode executors run on workers.

$SPARK_HOME/sbin/start-slave.sh spark://<host>::7077

That port 7077 is the master port. If master dies, then those workers lose
connection to port 7077 so I believe they go stale. So the spark-submit
carries on using the remaining executors on other workers.

So in summary one expects the job to run.  You start your UI on <HOST>:port.

One test you can do is to exit from UI and start UI on the host that
zookeeper selects the master on the same port. That should work.

HTH






Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 7 September 2016 at 15:27, arlindo santos <sa...@hotmail.com> wrote:

> Yes refreshed a few times. Running in cluster mode.
>
> Fyi.. I can duplicate this easily now. Our setup consists of 3 nodes
> running standalone spark, master and worker on each, zookeeper doing master
> leader election. If I kill a master on any node, the master shifts to
> another node and that is when the app state changes to waiting and never
> changes back to running on the gui, but really it's in a running mode.
>
> Sent from my BlackBerry 10 smartphone on the Rogers network.
> *From: *Mich Talebzadeh
> *Sent: *Wednesday, September 7, 2016 9:50 AM
> *To: *sarlindo
> *Cc: *user @spark
> *Subject: *Re: spark 1.6.0 web console shows running application in a
> "waiting" status, but it's acutally running
>
> Have you refreshed the Spark UI page?
>
> What Mode are you running your Spark app?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com> wrote:
>
>> I have 2 questions/issues.
>>
>> 1. We had the spark-master shut down (reason unknown) we looked at the
>> spark-master logs and it simply shows this, is there some other log I
>> should
>> be looking at to find out why the master went down?
>>
>> 16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
>> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181. Will not attempt to
>> authenticate using SASL (unknown error)
>> 16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
>> shutting down.
>> 16/09/05 21:10:00 INFO ClientCnxn: Socket connection established,
>> initiating
>> session, client: /142.201.219.75:56361, server:
>> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181
>>
>>
>> 2. Spark 1.6.0 web console shows a running application in a "waiting"
>> status, but it's actually running. Is this an existing bug?
>>
>> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-
>> application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>

Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by arlindo santos <sa...@hotmail.com>.
Yes refreshed a few times. Running in cluster mode.

Fyi.. I can duplicate this easily now. Our setup consists of 3 nodes running standalone spark, master and worker on each, zookeeper doing master leader election. If I kill a master on any node, the master shifts to another node and that is when the app state changes to waiting and never changes back to running on the gui, but really it's in a running mode.

Sent from my BlackBerry 10 smartphone on the Rogers network.
From: Mich Talebzadeh
Sent: Wednesday, September 7, 2016 9:50 AM
To: sarlindo
Cc: user @spark
Subject: Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running


Have you refreshed the Spark UI page?

What Mode are you running your Spark app?

HTH


Dr Mich Talebzadeh



LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.



On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com>> wrote:
I have 2 questions/issues.

1. We had the spark-master shut down (reason unknown) we looked at the
spark-master logs and it simply shows this, is there some other log I should
be looking at to find out why the master went down?

16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>. Will not attempt to
authenticate using SASL (unknown error)
16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
shutting down.
16/09/05 21:10:00 INFO ClientCnxn: Socket connection established, initiating
session, client: /142.201.219.75:56361<http://142.201.219.75:56361>, server:
tiplxapp-spk02.prd.tse.com/142.201.219.76:2181<http://tiplxapp-spk02.prd.tse.com/142.201.219.76:2181>


2. Spark 1.6.0 web console shows a running application in a "waiting"
status, but it's actually running. Is this an existing bug?

<http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png>





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>



Re: spark 1.6.0 web console shows running application in a "waiting" status, but it's acutally running

Posted by Mich Talebzadeh <mi...@gmail.com>.
Have you refreshed the Spark UI page?

What Mode are you running your Spark app?

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 6 September 2016 at 16:15, sarlindo <sa...@hotmail.com> wrote:

> I have 2 questions/issues.
>
> 1. We had the spark-master shut down (reason unknown) we looked at the
> spark-master logs and it simply shows this, is there some other log I
> should
> be looking at to find out why the master went down?
>
> 16/09/05 21:10:00 INFO ClientCnxn: Opening socket connection to server
> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 16/09/05 21:10:00 ERROR Master: Leadership has been revoked -- master
> shutting down.
> 16/09/05 21:10:00 INFO ClientCnxn: Socket connection established,
> initiating
> session, client: /142.201.219.75:56361, server:
> tiplxapp-spk02.prd.tse.com/142.201.219.76:2181
>
>
> 2. Spark 1.6.0 web console shows a running application in a "waiting"
> status, but it's actually running. Is this an existing bug?
>
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n27665/33.png>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/spark-1-6-0-web-console-shows-running-
> application-in-a-waiting-status-but-it-s-acutally-running-tp27665.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>