You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Dave Rugh <da...@gmail.com> on 2015/08/19 21:46:11 UTC
Role instance RoleInstance failed
I went through the Slider Memcached Tutorial
<http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_app.html>
and
was able to package/deploy/start the memcached container successfully;
however when I package up a custom application, basically a Java jar plus
dependencies, the container never launches succssfully.
The application page show the app is in a FINISHED/FAILED state with this
diagnostic:
http://quickstart.cloudera:8088/cluster/app/application_1439926335194_0001
Diagnostics: Unstable Application Instance : - failed with component
MYAPP failed 'recently' 6 times (4 in startup); threshold is 5 - last
failure: Failure container_1439926335194_0001_01_000008 on host
quickstart.cloudera (0):
http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:8041/container_1439926335
<http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:8041/container_1439926335194_0001_01_000008/ctx/C4>
Looking at the container logs - nothing is in the slider-out.txt file
indicating that the container never ran; no errors in the slider-err.txt
file (see below) and the slider-log.txt file is quite lengthy but the only
error I see is this:
2015-08-19 12:14:05,744 [AMRM Callback Handler Thread] ERROR
appmaster.SliderAppMaster - Role instance RoleInstance{role='MYAPP',
id='container_1439926335194_0003_01_000002',
container=ContainerID=container_1439926335194_0003_01_000002
nodeID=quickstart.cloudera:8041 http=quickstart.cloudera:8042
priority=1073741825 resource=<memory:1024, vCores:1>,
createTime=1440011595168, startTime=1440011595313, released=false,
roleId=1, host=quickstart.cloudera, hostURL=http://quickstart.cloudera:8042,
state=5, placement=null, exitCode=0, command='python
./infra/agent/slider-agent/agent/main.py --label
container_1439926335194_0003_01_000002___MYAPP --zk-quorum localhost:2181
--zk-reg-path /registry/users/myuser/services/org-apache-slider/myapp1 >
<LOG_DIR>/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
environment=[LANGUAGE="en_US.UTF-8", AGENT_WORK_ROOT="$PWD",
HADOOP_USER_NAME="MYUSER", AGENT_LOG_ROOT="<LOG_DIR>",
PYTHONPATH="./infra/agent/slider-agent/", LC_ALL="en_US.UTF-8",
SLIDER_PASSPHRASE="<redacted>", LANG="en_US.UTF-8"]} failed
What's my next step to debug this?
Do I want to attach to the code that launches the container and step
through? I have Netbeans available in the environment.
Thanks,
Dave
*slider-err.txt*
*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.PackagesResourceConfig
init*
*INFO: Scanning for root resource and provider classes in the packages:*
* org.apache.slider.server.appmaster.web.rest.agent*
*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
logClasses*
*INFO: Root resource classes found:*
* class org.apache.slider.server.appmaster.web.rest.agent.AgentWebServices*
*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
init*
*INFO: No provider classes found.*
*Aug 19, 2015 12:13:07 PM
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate*
*INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17
AM'*
*Aug 19, 2015 12:13:18 PM com.sun.jersey.spi.inject.Errors
processErrorMessages*
*WARNING: The following warnings have been detected with resource and/or
provider classes:*
* WARNING: A sub-resource method, public javax.ws.rs.core.Response
org.apache.slider.server.appmaster.web.rest.agent.AgentResource.endpointRoot(),
with URI template, "/", is treated as a resource method*
Re: Role instance RoleInstance failed
Posted by Gour Saha <gs...@hortonworks.com>.
Great. Let us know if you hit any other issues.
-Gour
On 8/19/15, 3:26 PM, "Dave Rugh" <da...@gmail.com> wrote:
>I found the container logs via the containers web UI (on Cloudera VM it is
>http://quickstart.cloudera:8042/node/allContainers)
>
>There are 2 containers for my application, first just shows the logs I was
>looking at earlier indicating whether the container succeeded or failed;
>second has many logs with useful info (command / errors / slider-agent /
>status_command).
>
>They are transient, but I was able to look at them before the application
>terminated.
>
>slider-agent.out just has this line in it:
>
>*No handlers could be found for logger "root"*
>
>However *slider-agent.log* gave me the info I was looking for, basically
>the stderr / stdout from executing the Java command line so that is very
>helpful.
>
>INFO 2015-08-19 14:07:28,422 AgentToggleLogger.py:40 - Queue result:
>{'componentStatus': [],
> 'reports': [{'actionId': u'4-1',
> 'clusterName': u'myapp1',
> 'exitcode': 1,
> 'reportResult': True,
> 'role': u'MYAPP',
> 'roleCommand': u'START',
> 'serviceName': u'myapp1',
> 'status': 'FAILED',
> 'stderr': '2015-08-19 14:07:28,268 - Error while executing
>command ...<removed for brevity>,
> 'stdout': '2015-08-19 14:07:23,261 -
>Execute[\'/usr/java/latest/bin/java -Xmx256m -classpath ...<removed for
>brevity>,
> 'structuredOut': '{}',
> 'taskId': 4}]}
>
>
>Thanks, put me on the path to solving this.
>
>Dave
>
>
>
>
>On Wed, Aug 19, 2015 at 12:53 PM, Gour Saha <gs...@hortonworks.com> wrote:
>
>> Can you find the slider-agent.out available in the container logs for
>> container_1439926335194_0003_01_000002 and see what¹s in there?
>>
>> -Gour
>>
>> On 8/19/15, 12:46 PM, "Dave Rugh" <da...@gmail.com> wrote:
>>
>> >I went through the Slider Memcached Tutorial
>> ><
>>
>>http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_a
>> >pp.html>
>> >and
>> >was able to package/deploy/start the memcached container successfully;
>> >however when I package up a custom application, basically a Java jar
>>plus
>> >dependencies, the container never launches succssfully.
>> >
>> >
>> >
>> >The application page show the app is in a FINISHED/FAILED state with
>>this
>> >diagnostic:
>> >
>> >
>>
>>http://quickstart.cloudera:8088/cluster/app/application_1439926335194_000
>>1
>> >
>> >
>> >
>> >Diagnostics: Unstable Application Instance : - failed with component
>> >MYAPP failed 'recently' 6 times (4 in startup); threshold is 5 - last
>> >failure: Failure container_1439926335194_0001_01_000008 on host
>> >quickstart.cloudera (0):
>> >
>>
>>http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:804
>>1
>> >/container_1439926335
>> ><
>>
>>http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:804
>> >1/container_1439926335194_0001_01_000008/ctx/C4>
>> >
>> >
>> >Looking at the container logs - nothing is in the slider-out.txt file
>> >indicating that the container never ran; no errors in the
>>slider-err.txt
>> >file (see below) and the slider-log.txt file is quite lengthy but the
>>only
>> >error I see is this:
>> >
>> >
>> >
>> >2015-08-19 12:14:05,744 [AMRM Callback Handler Thread] ERROR
>> >appmaster.SliderAppMaster - Role instance RoleInstance{role='MYAPP',
>> >id='container_1439926335194_0003_01_000002',
>> >container=ContainerID=container_1439926335194_0003_01_000002
>> >nodeID=quickstart.cloudera:8041 http=quickstart.cloudera:8042
>> >priority=1073741825 resource=<memory:1024, vCores:1>,
>> >createTime=1440011595168, startTime=1440011595313, released=false,
>> >roleId=1, host=quickstart.cloudera,
>> >hostURL=http://quickstart.cloudera:8042,
>> >state=5, placement=null, exitCode=0, command='python
>> >./infra/agent/slider-agent/agent/main.py --label
>> >container_1439926335194_0003_01_000002___MYAPP --zk-quorum
>>localhost:2181
>> >--zk-reg-path /registry/users/myuser/services/org-apache-slider/myapp1
>>>
>> ><LOG_DIR>/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
>> >environment=[LANGUAGE="en_US.UTF-8", AGENT_WORK_ROOT="$PWD",
>> >HADOOP_USER_NAME="MYUSER", AGENT_LOG_ROOT="<LOG_DIR>",
>> >PYTHONPATH="./infra/agent/slider-agent/", LC_ALL="en_US.UTF-8",
>> >SLIDER_PASSPHRASE="<redacted>", LANG="en_US.UTF-8"]} failed
>> >
>> >
>> >What's my next step to debug this?
>> >
>> >
>> >Do I want to attach to the code that launches the container and step
>> >through? I have Netbeans available in the environment.
>> >
>> >
>> >Thanks,
>> >
>> >
>> >Dave
>> >
>> >
>> >
>> >*slider-err.txt*
>> >
>> >
>> >*Aug 19, 2015 12:13:07 PM
>>com.sun.jersey.api.core.PackagesResourceConfig
>> >init*
>> >
>> >*INFO: Scanning for root resource and provider classes in the
>>packages:*
>> >
>> >* org.apache.slider.server.appmaster.web.rest.agent*
>> >
>> >*Aug 19, 2015 12:13:07 PM
>>com.sun.jersey.api.core.ScanningResourceConfig
>> >logClasses*
>> >
>> >*INFO: Root resource classes found:*
>> >
>> >* class
>> >org.apache.slider.server.appmaster.web.rest.agent.AgentWebServices*
>> >
>> >*Aug 19, 2015 12:13:07 PM
>>com.sun.jersey.api.core.ScanningResourceConfig
>> >init*
>> >
>> >*INFO: No provider classes found.*
>> >
>> >*Aug 19, 2015 12:13:07 PM
>> >com.sun.jersey.server.impl.application.WebApplicationImpl _initiate*
>> >
>> >*INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
>> >11:17
>> >AM'*
>> >
>> >*Aug 19, 2015 12:13:18 PM com.sun.jersey.spi.inject.Errors
>> >processErrorMessages*
>> >
>> >*WARNING: The following warnings have been detected with resource
>>and/or
>> >provider classes:*
>> >
>> >* WARNING: A sub-resource method, public javax.ws.rs.core.Response
>>
>>>org.apache.slider.server.appmaster.web.rest.agent.AgentResource.endpoint
>>>Ro
>> >ot(),
>> >with URI template, "/", is treated as a resource method*
>>
>>
Re: Role instance RoleInstance failed
Posted by Dave Rugh <da...@gmail.com>.
I found the container logs via the containers web UI (on Cloudera VM it is
http://quickstart.cloudera:8042/node/allContainers)
There are 2 containers for my application, first just shows the logs I was
looking at earlier indicating whether the container succeeded or failed;
second has many logs with useful info (command / errors / slider-agent /
status_command).
They are transient, but I was able to look at them before the application
terminated.
slider-agent.out just has this line in it:
*No handlers could be found for logger "root"*
However *slider-agent.log* gave me the info I was looking for, basically
the stderr / stdout from executing the Java command line so that is very
helpful.
INFO 2015-08-19 14:07:28,422 AgentToggleLogger.py:40 - Queue result:
{'componentStatus': [],
'reports': [{'actionId': u'4-1',
'clusterName': u'myapp1',
'exitcode': 1,
'reportResult': True,
'role': u'MYAPP',
'roleCommand': u'START',
'serviceName': u'myapp1',
'status': 'FAILED',
'stderr': '2015-08-19 14:07:28,268 - Error while executing
command ...<removed for brevity>,
'stdout': '2015-08-19 14:07:23,261 -
Execute[\'/usr/java/latest/bin/java -Xmx256m -classpath ...<removed for
brevity>,
'structuredOut': '{}',
'taskId': 4}]}
Thanks, put me on the path to solving this.
Dave
On Wed, Aug 19, 2015 at 12:53 PM, Gour Saha <gs...@hortonworks.com> wrote:
> Can you find the slider-agent.out available in the container logs for
> container_1439926335194_0003_01_000002 and see what¹s in there?
>
> -Gour
>
> On 8/19/15, 12:46 PM, "Dave Rugh" <da...@gmail.com> wrote:
>
> >I went through the Slider Memcached Tutorial
> ><
> http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_a
> >pp.html>
> >and
> >was able to package/deploy/start the memcached container successfully;
> >however when I package up a custom application, basically a Java jar plus
> >dependencies, the container never launches succssfully.
> >
> >
> >
> >The application page show the app is in a FINISHED/FAILED state with this
> >diagnostic:
> >
> >
> http://quickstart.cloudera:8088/cluster/app/application_1439926335194_0001
> >
> >
> >
> >Diagnostics: Unstable Application Instance : - failed with component
> >MYAPP failed 'recently' 6 times (4 in startup); threshold is 5 - last
> >failure: Failure container_1439926335194_0001_01_000008 on host
> >quickstart.cloudera (0):
> >
> http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:8041
> >/container_1439926335
> ><
> http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:804
> >1/container_1439926335194_0001_01_000008/ctx/C4>
> >
> >
> >Looking at the container logs - nothing is in the slider-out.txt file
> >indicating that the container never ran; no errors in the slider-err.txt
> >file (see below) and the slider-log.txt file is quite lengthy but the only
> >error I see is this:
> >
> >
> >
> >2015-08-19 12:14:05,744 [AMRM Callback Handler Thread] ERROR
> >appmaster.SliderAppMaster - Role instance RoleInstance{role='MYAPP',
> >id='container_1439926335194_0003_01_000002',
> >container=ContainerID=container_1439926335194_0003_01_000002
> >nodeID=quickstart.cloudera:8041 http=quickstart.cloudera:8042
> >priority=1073741825 resource=<memory:1024, vCores:1>,
> >createTime=1440011595168, startTime=1440011595313, released=false,
> >roleId=1, host=quickstart.cloudera,
> >hostURL=http://quickstart.cloudera:8042,
> >state=5, placement=null, exitCode=0, command='python
> >./infra/agent/slider-agent/agent/main.py --label
> >container_1439926335194_0003_01_000002___MYAPP --zk-quorum localhost:2181
> >--zk-reg-path /registry/users/myuser/services/org-apache-slider/myapp1 >
> ><LOG_DIR>/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
> >environment=[LANGUAGE="en_US.UTF-8", AGENT_WORK_ROOT="$PWD",
> >HADOOP_USER_NAME="MYUSER", AGENT_LOG_ROOT="<LOG_DIR>",
> >PYTHONPATH="./infra/agent/slider-agent/", LC_ALL="en_US.UTF-8",
> >SLIDER_PASSPHRASE="<redacted>", LANG="en_US.UTF-8"]} failed
> >
> >
> >What's my next step to debug this?
> >
> >
> >Do I want to attach to the code that launches the container and step
> >through? I have Netbeans available in the environment.
> >
> >
> >Thanks,
> >
> >
> >Dave
> >
> >
> >
> >*slider-err.txt*
> >
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.PackagesResourceConfig
> >init*
> >
> >*INFO: Scanning for root resource and provider classes in the packages:*
> >
> >* org.apache.slider.server.appmaster.web.rest.agent*
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
> >logClasses*
> >
> >*INFO: Root resource classes found:*
> >
> >* class
> >org.apache.slider.server.appmaster.web.rest.agent.AgentWebServices*
> >
> >*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
> >init*
> >
> >*INFO: No provider classes found.*
> >
> >*Aug 19, 2015 12:13:07 PM
> >com.sun.jersey.server.impl.application.WebApplicationImpl _initiate*
> >
> >*INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
> >11:17
> >AM'*
> >
> >*Aug 19, 2015 12:13:18 PM com.sun.jersey.spi.inject.Errors
> >processErrorMessages*
> >
> >*WARNING: The following warnings have been detected with resource and/or
> >provider classes:*
> >
> >* WARNING: A sub-resource method, public javax.ws.rs.core.Response
> >org.apache.slider.server.appmaster.web.rest.agent.AgentResource.endpointRo
> >ot(),
> >with URI template, "/", is treated as a resource method*
>
>
Re: Role instance RoleInstance failed
Posted by Gour Saha <gs...@hortonworks.com>.
Can you find the slider-agent.out available in the container logs for
container_1439926335194_0003_01_000002 and see what¹s in there?
-Gour
On 8/19/15, 12:46 PM, "Dave Rugh" <da...@gmail.com> wrote:
>I went through the Slider Memcached Tutorial
><http://slider.incubator.apache.org/docs/slider_specs/hello_world_slider_a
>pp.html>
>and
>was able to package/deploy/start the memcached container successfully;
>however when I package up a custom application, basically a Java jar plus
>dependencies, the container never launches succssfully.
>
>
>
>The application page show the app is in a FINISHED/FAILED state with this
>diagnostic:
>
>http://quickstart.cloudera:8088/cluster/app/application_1439926335194_0001
>
>
>
>Diagnostics: Unstable Application Instance : - failed with component
>MYAPP failed 'recently' 6 times (4 in startup); threshold is 5 - last
>failure: Failure container_1439926335194_0001_01_000008 on host
>quickstart.cloudera (0):
>http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:8041
>/container_1439926335
><http://quickstart.cloudera:19888/jobhistory/logs//quickstart.cloudera:804
>1/container_1439926335194_0001_01_000008/ctx/C4>
>
>
>Looking at the container logs - nothing is in the slider-out.txt file
>indicating that the container never ran; no errors in the slider-err.txt
>file (see below) and the slider-log.txt file is quite lengthy but the only
>error I see is this:
>
>
>
>2015-08-19 12:14:05,744 [AMRM Callback Handler Thread] ERROR
>appmaster.SliderAppMaster - Role instance RoleInstance{role='MYAPP',
>id='container_1439926335194_0003_01_000002',
>container=ContainerID=container_1439926335194_0003_01_000002
>nodeID=quickstart.cloudera:8041 http=quickstart.cloudera:8042
>priority=1073741825 resource=<memory:1024, vCores:1>,
>createTime=1440011595168, startTime=1440011595313, released=false,
>roleId=1, host=quickstart.cloudera,
>hostURL=http://quickstart.cloudera:8042,
>state=5, placement=null, exitCode=0, command='python
>./infra/agent/slider-agent/agent/main.py --label
>container_1439926335194_0003_01_000002___MYAPP --zk-quorum localhost:2181
>--zk-reg-path /registry/users/myuser/services/org-apache-slider/myapp1 >
><LOG_DIR>/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
>environment=[LANGUAGE="en_US.UTF-8", AGENT_WORK_ROOT="$PWD",
>HADOOP_USER_NAME="MYUSER", AGENT_LOG_ROOT="<LOG_DIR>",
>PYTHONPATH="./infra/agent/slider-agent/", LC_ALL="en_US.UTF-8",
>SLIDER_PASSPHRASE="<redacted>", LANG="en_US.UTF-8"]} failed
>
>
>What's my next step to debug this?
>
>
>Do I want to attach to the code that launches the container and step
>through? I have Netbeans available in the environment.
>
>
>Thanks,
>
>
>Dave
>
>
>
>*slider-err.txt*
>
>
>*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.PackagesResourceConfig
>init*
>
>*INFO: Scanning for root resource and provider classes in the packages:*
>
>* org.apache.slider.server.appmaster.web.rest.agent*
>
>*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
>logClasses*
>
>*INFO: Root resource classes found:*
>
>* class
>org.apache.slider.server.appmaster.web.rest.agent.AgentWebServices*
>
>*Aug 19, 2015 12:13:07 PM com.sun.jersey.api.core.ScanningResourceConfig
>init*
>
>*INFO: No provider classes found.*
>
>*Aug 19, 2015 12:13:07 PM
>com.sun.jersey.server.impl.application.WebApplicationImpl _initiate*
>
>*INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011
>11:17
>AM'*
>
>*Aug 19, 2015 12:13:18 PM com.sun.jersey.spi.inject.Errors
>processErrorMessages*
>
>*WARNING: The following warnings have been detected with resource and/or
>provider classes:*
>
>* WARNING: A sub-resource method, public javax.ws.rs.core.Response
>org.apache.slider.server.appmaster.web.rest.agent.AgentResource.endpointRo
>ot(),
>with URI template, "/", is treated as a resource method*