You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by akhandeshi <am...@gmail.com> on 2015/02/02 16:56:08 UTC
"Loading" status
I am not sure what Loading status means, followed by Running. In the
application UI, I see:
Executor Summary
ExecutorID Worker Cores Memory State Logs
1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874 16 83971
LOADING stdout stderr
0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685 16 83971
RUNNING stdout stderr
Looking at the executor hadoop-w-2, I see the status is "Loading" . Why
different statuses, and what does that mean?
Please see below for details:
ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
Master URL: spark://hadoop-m:7077
Cores: 16 (16 Used)
Memory: 82.0 GB (82.0 GB Used)
Back to Master
Running Executors (1)
ExecutorID Cores State Memory Job Details Logs
0 16 LOADING 82.0 GB
ID: app-20150202152154-0001
Name: Simple File Merge Application
User: hadoop
stdout stderr
Thank you,
Ami
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: "Loading" status
Posted by Ami Khandeshi <am...@gmail.com>.
It seems sort of Listener UI error! I say this because, I see the status
in the executor web UI to be loading, but the application UI, for the
same executor the status is Running! I have also seen the reverse behavior
where the application UI indicates a particular executor as loading, but
the executor UI page indicates it is running! There are no error messages
in the logs that I have been able to spot! Also, logs seems to indicate it
is doing the work..
As far as reproducibility goes, it is hard, due to its inconsistent
behavior observed...perhaps, again indicating some listener issue...
will update as I find any further information...
thank you both for help!
On Monday, February 2, 2015, Mark Hamstra <ma...@clearstorydata.com> wrote:
> Yes, if the Master is unable to register the Executor and transition it to
> RUNNING, then the Executor will stay in LOADING state, so this can be
> caused by problems in the Master or the Master-Executor communication.
>
>
> On Mon, Feb 2, 2015 at 9:24 AM, Tushar Sharma <tusharshar@gmail.com
> <javascript:_e(%7B%7D,'cvml','tusharshar@gmail.com');>> wrote:
>
>> Yes curious indeed. Usually if such status persists it leads to executor
>> failure after a couple of tries. We also noticed similar behavior but the
>> logs clearly mentioned the lack of some akka resource as the reason. I am
>> sure the logs also holds the key for this particular problem.
>>
>> On Mon, Feb 2, 2015 at 10:49 PM, Mark Hamstra <mark@clearstorydata.com
>> <javascript:_e(%7B%7D,'cvml','mark@clearstorydata.com');>> wrote:
>>
>>> Curious. I guess the first question is whether we've got some sort of
>>> Listener/UI error so that the UI is not accurately reflecting the
>>> Executor's actual state, or whether the "LOADING" Executor really is
>>> spending a considerable length of time in this "I'm in the process of being
>>> created, but not yet doing anything useful" state.
>>>
>>> If you can figure out a little more of what is going on or how to
>>> reproduce this state, please do file a JIRA.
>>>
>>>
>>> On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi <ami.khandeshi@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','ami.khandeshi@gmail.com');>> wrote:
>>>
>>>> Yes
>>>>
>>>>
>>>> On Monday, February 2, 2015, Mark Hamstra <mark@clearstorydata.com
>>>> <javascript:_e(%7B%7D,'cvml','mark@clearstorydata.com');>> wrote:
>>>>
>>>>> LOADING is just the state in which new Executors are created but
>>>>> before they have everything they need and are fully registered to
>>>>> transition to state RUNNING and begin doing actual work:
>>>>>
>>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351
>>>>>
>>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133
>>>>>
>>>>> LOADING should be a fairly brief, transitory state. Are you seeing
>>>>> Executors remaining in LOADING for a significant length of time?
>>>>>
>>>>>
>>>>> On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi <am...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am not sure what Loading status means, followed by Running. In the
>>>>>> application UI, I see:
>>>>>> Executor Summary
>>>>>>
>>>>>> ExecutorID Worker Cores Memory State Logs
>>>>>> 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874
>>>>>> 16 83971
>>>>>> LOADING stdout stderr
>>>>>> 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>>>>> 16 83971
>>>>>> RUNNING stdout stderr
>>>>>>
>>>>>> Looking at the executor hadoop-w-2, I see the status is "Loading" .
>>>>>> Why
>>>>>> different statuses, and what does that mean?
>>>>>>
>>>>>> Please see below for details:
>>>>>>
>>>>>> ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>>>>> Master URL: spark://hadoop-m:7077
>>>>>> Cores: 16 (16 Used)
>>>>>> Memory: 82.0 GB (82.0 GB Used)
>>>>>> Back to Master
>>>>>>
>>>>>> Running Executors (1)
>>>>>>
>>>>>> ExecutorID Cores State Memory Job Details Logs
>>>>>> 0 16 LOADING 82.0 GB
>>>>>> ID: app-20150202152154-0001
>>>>>> Name: Simple File Merge Application
>>>>>> User: hadoop
>>>>>> stdout stderr
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Ami
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>> Nabble.com.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: "Loading" status
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Yes, if the Master is unable to register the Executor and transition it to
RUNNING, then the Executor will stay in LOADING state, so this can be
caused by problems in the Master or the Master-Executor communication.
On Mon, Feb 2, 2015 at 9:24 AM, Tushar Sharma <tu...@gmail.com> wrote:
> Yes curious indeed. Usually if such status persists it leads to executor
> failure after a couple of tries. We also noticed similar behavior but the
> logs clearly mentioned the lack of some akka resource as the reason. I am
> sure the logs also holds the key for this particular problem.
>
> On Mon, Feb 2, 2015 at 10:49 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
>> Curious. I guess the first question is whether we've got some sort of
>> Listener/UI error so that the UI is not accurately reflecting the
>> Executor's actual state, or whether the "LOADING" Executor really is
>> spending a considerable length of time in this "I'm in the process of being
>> created, but not yet doing anything useful" state.
>>
>> If you can figure out a little more of what is going on or how to
>> reproduce this state, please do file a JIRA.
>>
>>
>> On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi <am...@gmail.com>
>> wrote:
>>
>>> Yes
>>>
>>>
>>> On Monday, February 2, 2015, Mark Hamstra <ma...@clearstorydata.com>
>>> wrote:
>>>
>>>> LOADING is just the state in which new Executors are created but before
>>>> they have everything they need and are fully registered to transition to
>>>> state RUNNING and begin doing actual work:
>>>>
>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351
>>>>
>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133
>>>>
>>>> LOADING should be a fairly brief, transitory state. Are you seeing
>>>> Executors remaining in LOADING for a significant length of time?
>>>>
>>>>
>>>> On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi <am...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am not sure what Loading status means, followed by Running. In the
>>>>> application UI, I see:
>>>>> Executor Summary
>>>>>
>>>>> ExecutorID Worker Cores Memory State Logs
>>>>> 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874
>>>>> 16 83971
>>>>> LOADING stdout stderr
>>>>> 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>>>> 16 83971
>>>>> RUNNING stdout stderr
>>>>>
>>>>> Looking at the executor hadoop-w-2, I see the status is "Loading" .
>>>>> Why
>>>>> different statuses, and what does that mean?
>>>>>
>>>>> Please see below for details:
>>>>>
>>>>> ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>>>> Master URL: spark://hadoop-m:7077
>>>>> Cores: 16 (16 Used)
>>>>> Memory: 82.0 GB (82.0 GB Used)
>>>>> Back to Master
>>>>>
>>>>> Running Executors (1)
>>>>>
>>>>> ExecutorID Cores State Memory Job Details Logs
>>>>> 0 16 LOADING 82.0 GB
>>>>> ID: app-20150202152154-0001
>>>>> Name: Simple File Merge Application
>>>>> User: hadoop
>>>>> stdout stderr
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Ami
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>
>
Re: "Loading" status
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Curious. I guess the first question is whether we've got some sort of
Listener/UI error so that the UI is not accurately reflecting the
Executor's actual state, or whether the "LOADING" Executor really is
spending a considerable length of time in this "I'm in the process of being
created, but not yet doing anything useful" state.
If you can figure out a little more of what is going on or how to reproduce
this state, please do file a JIRA.
On Mon, Feb 2, 2015 at 8:28 AM, Ami Khandeshi <am...@gmail.com>
wrote:
> Yes
>
>
> On Monday, February 2, 2015, Mark Hamstra <ma...@clearstorydata.com> wrote:
>
>> LOADING is just the state in which new Executors are created but before
>> they have everything they need and are fully registered to transition to
>> state RUNNING and begin doing actual work:
>>
>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351
>>
>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133
>>
>> LOADING should be a fairly brief, transitory state. Are you seeing
>> Executors remaining in LOADING for a significant length of time?
>>
>>
>> On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi <am...@gmail.com>
>> wrote:
>>
>>> I am not sure what Loading status means, followed by Running. In the
>>> application UI, I see:
>>> Executor Summary
>>>
>>> ExecutorID Worker Cores Memory State Logs
>>> 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874
>>> 16 83971
>>> LOADING stdout stderr
>>> 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>> 16 83971
>>> RUNNING stdout stderr
>>>
>>> Looking at the executor hadoop-w-2, I see the status is "Loading" . Why
>>> different statuses, and what does that mean?
>>>
>>> Please see below for details:
>>>
>>> ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>>> Master URL: spark://hadoop-m:7077
>>> Cores: 16 (16 Used)
>>> Memory: 82.0 GB (82.0 GB Used)
>>> Back to Master
>>>
>>> Running Executors (1)
>>>
>>> ExecutorID Cores State Memory Job Details Logs
>>> 0 16 LOADING 82.0 GB
>>> ID: app-20150202152154-0001
>>> Name: Simple File Merge Application
>>> User: hadoop
>>> stdout stderr
>>>
>>> Thank you,
>>>
>>> Ami
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
Re: "Loading" status
Posted by Ami Khandeshi <am...@gmail.com>.
Yes
On Monday, February 2, 2015, Mark Hamstra <ma...@clearstorydata.com> wrote:
> LOADING is just the state in which new Executors are created but before
> they have everything they need and are fully registered to transition to
> state RUNNING and begin doing actual work:
>
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351
>
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133
>
> LOADING should be a fairly brief, transitory state. Are you seeing
> Executors remaining in LOADING for a significant length of time?
>
>
> On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi <ami.khandeshi@gmail.com
> <javascript:_e(%7B%7D,'cvml','ami.khandeshi@gmail.com');>> wrote:
>
>> I am not sure what Loading status means, followed by Running. In the
>> application UI, I see:
>> Executor Summary
>>
>> ExecutorID Worker Cores Memory State Logs
>> 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874
>> 16 83971
>> LOADING stdout stderr
>> 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>> 16 83971
>> RUNNING stdout stderr
>>
>> Looking at the executor hadoop-w-2, I see the status is "Loading" . Why
>> different statuses, and what does that mean?
>>
>> Please see below for details:
>>
>> ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
>> Master URL: spark://hadoop-m:7077
>> Cores: 16 (16 Used)
>> Memory: 82.0 GB (82.0 GB Used)
>> Back to Master
>>
>> Running Executors (1)
>>
>> ExecutorID Cores State Memory Job Details Logs
>> 0 16 LOADING 82.0 GB
>> ID: app-20150202152154-0001
>> Name: Simple File Merge Application
>> User: hadoop
>> stdout stderr
>>
>> Thank you,
>>
>> Ami
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> <javascript:_e(%7B%7D,'cvml','user-unsubscribe@spark.apache.org');>
>> For additional commands, e-mail: user-help@spark.apache.org
>> <javascript:_e(%7B%7D,'cvml','user-help@spark.apache.org');>
>>
>>
>
Re: "Loading" status
Posted by Mark Hamstra <ma...@clearstorydata.com>.
LOADING is just the state in which new Executors are created but before
they have everything they need and are fully registered to transition to
state RUNNING and begin doing actual work:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L351
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala#L133
LOADING should be a fairly brief, transitory state. Are you seeing
Executors remaining in LOADING for a significant length of time?
On Mon, Feb 2, 2015 at 7:56 AM, akhandeshi <am...@gmail.com> wrote:
> I am not sure what Loading status means, followed by Running. In the
> application UI, I see:
> Executor Summary
>
> ExecutorID Worker Cores Memory State Logs
> 1 worker-20150202144112-hadoop-w-1.c.fi-mdd-poc.internal-38874
> 16 83971
> LOADING stdout stderr
> 0 worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
> 16 83971
> RUNNING stdout stderr
>
> Looking at the executor hadoop-w-2, I see the status is "Loading" . Why
> different statuses, and what does that mean?
>
> Please see below for details:
>
> ID: worker-20150202144112-hadoop-w-2.c.fi-mdd-poc.internal-58685
> Master URL: spark://hadoop-m:7077
> Cores: 16 (16 Used)
> Memory: 82.0 GB (82.0 GB Used)
> Back to Master
>
> Running Executors (1)
>
> ExecutorID Cores State Memory Job Details Logs
> 0 16 LOADING 82.0 GB
> ID: app-20150202152154-0001
> Name: Simple File Merge Application
> User: hadoop
> stdout stderr
>
> Thank you,
>
> Ami
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Loading-status-tp21468.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>