You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/03/25 21:28:55 UTC
[jira] [Resolved] (SLIDER-646) AgentLaunchFailureIT test failing at
times
[ https://issues.apache.org/jira/browse/SLIDER-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved SLIDER-646.
-----------------------------------
Resolution: Fixed
Fix Version/s: Slider 0.80
> AgentLaunchFailureIT test failing at times
> ------------------------------------------
>
> Key: SLIDER-646
> URL: https://issues.apache.org/jira/browse/SLIDER-646
> Project: Slider
> Issue Type: Bug
> Components: test
> Reporter: Gour Saha
> Assignee: Steve Loughran
> Fix For: Slider 0.80, Slider 0.60
>
>
> Chaos Monkey initial delay should be deterministic. It is currently set to 60 seconds. Subsequent interval is also set to 60 secs. However AgentLaunchFailureIT fails at times because the AM does not get sufficient time to startup. In one failure scenario it has been seen to fail within 300 ms of Chaos Monkey setup. This test fails about once in every 10 attempts.
> Here is the test output -
> {code}
> ------------------------------------------------------------------------------
> Test set: org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 43.503 sec <<< FAILURE! - in org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
> testAgentLaunchFailure(org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT) Time elapsed: 40.903 sec <<< FAILURE!
> java.lang.AssertionError: Application Launch Failure, exit code 68
> Chaos monkey triggered launch failure
> at org.junit.Assert.fail(Assert.java:88)
> at org.apache.slider.funtest.framework.CommandTestBase.createTemplatedSliderApplication(CommandTestBase.groovy:676)
> at org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT.testAgentLaunchFailure(AgentLaunchFailureIT.groovy:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
> Here is the AM log snippet -
> {code}
> 2014-11-12 09:29:34,989 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(764)) - Token YARN_AM_RM_TOKEN
> 2014-11-12 09:29:34,990 [main] INFO agent.AgentUtils (AgentUtils.java:getApplicationMetainfo(43)) - Reading metainfo at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,014 [main] INFO tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading metainfo.xml of size 1995
> 2014-11-12 09:29:35,096 [main] INFO agent.AgentUtils (AgentUtils.java:getDefaultConfig(64)) - Reading default config file configuration/cl-site.xml at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,102 [main] INFO tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading configuration/cl-site.xml of size 1270
> 2014-11-12 09:29:35,106 [main] INFO agent.HeartbeatMonitor (HeartbeatMonitor.java:start(46)) - Starting heartbeat monitor with interval 60000
> 2014-11-12 09:29:35,107 [Thread-36] DEBUG agent.HeartbeatMonitor (HeartbeatMonitor.java:run(65)) - Putting monitor to sleep for 60000 milliseconds
> 2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:buildInstance(502)) - Adding role COMMAND_LOGGER
> 2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:createDynamicProviderRole(585)) - Role COMMAND_LOGGER assigned priority 1
> 2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
> 2014-11-12 09:29:35,253 [main] DEBUG state.RoleHistory (RoleHistory.java:onBootstrap(370)) - Role history bootstrapped
> 2014-11-12 09:29:35,268 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2183)) - Adding Chaos Monkey scheduled every 60 seconds (0 hours -delay 60
> 2014-11-12 09:29:35,269 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2195)) - Chaos Monkey has triggered AM Launch failure
> 2014-11-12 09:29:35,269 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
> 2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action AM killer not enabled
> 2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action Container killer not enabled
> 2014-11-12 09:29:35,270 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2222)) - Chaos monkey not started
> 2014-11-12 09:29:35,271 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(879)) - HADOOP_USER_NAME='yarn'
> 2014-11-12 09:29:35,287 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(882)) - Registry service username =yarn
> 2014-11-12 09:29:35,308 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/yarn-site.xml
> 2014-11-12 09:29:35,342 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/core-site.xml
> 2014-11-12 09:29:35,375 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/hdfs-site.xml
> 2014-11-12 09:29:35,437 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1083)) - Service Record
> ServiceRecord{description='Slider Application Master'; external endpoints: {{
> "api" : "http://",
> "addressType" : "uri",
> "protocolType" : "webui",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.management",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.registry",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.configurations",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.exports",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
> } ]
> }; }; internal endpoints: {{
> "api" : "classpath:org.apache.slider.agents.secure",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.agents.oneway",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
> } ]
> }; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
> 2014-11-12 09:29:35,473 [main] INFO zk.RegistryOperationsService (RegistryOperationsService.java:bind(110)) - Bound at /users/yarn/services/org-apache-slider/test-agent-launchfail : ServiceRecord{description='Slider Application Master'; external endpoints: {{
> "api" : "http://",
> "addressType" : "uri",
> "protocolType" : "webui",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.management",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.registry",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.configurations",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.publisher.exports",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
> } ]
> }; }; internal endpoints: {{
> "api" : "classpath:org.apache.slider.agents.secure",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
> } ]
> }; {
> "api" : "classpath:org.apache.slider.agents.oneway",
> "addressType" : "uri",
> "protocolType" : "REST",
> "addresses" : [ {
> "uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
> } ]
> }; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
> 2014-11-12 09:29:35,535 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1085)) - Registered service under /users/yarn/services/org-apache-slider/test-agent-launchfail; absolute path /registry/users/yarn/services/org-apache-slider/test-agent-launchfail
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(888)) - RM Webapp address 172.31.11.99:8088
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(889)) - slider Webapp address http://172.31.11.97:37091
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(892)) - Application Master Initialization Completed
> 2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:startQueueProcessing(463)) - Queue Processing started
> 2014-11-12 09:29:35,544 [AmExecutor-005] INFO actions.QueueService (QueueService.java:run(171)) - QueueService processor started
> 2014-11-12 09:29:35,545 [AmExecutor-006] INFO actions.QueueExecutor (QueueExecutor.java:run(68)) - Queue Executor run() started
> 2014-11-12 09:29:35,545 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
> 2014-11-12 09:29:35,545 [AmExecutor-006] INFO appmaster.SliderAppMaster (ActionStopSlider.java:execute(118)) - SliderAppMasterApi.stopCluster: Chaos monkey triggered launch failure
> 2014-11-12 09:29:35,619 [main] DEBUG agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(117)) - Validating conf {,
> "internal": {
> "schema" : "http://example.org/specification/v2.0.0",
> "metadata" : {
> "create.hadoop.deployed.info" : "(no branch) @ae493e14a5f8a78bd6227e6d377bbef6",
> "create.application.build.info" : "Slider Core-0.60.0.2.2.0.0-1947 Built against commit# ${buildNumber} on Java 1.7.0_67 by yarn",
> "create.hadoop.build.info" : "2.6.0.2.2.0.0-1947",
> "create.time.millis" : "1415784562671",
> "create.time" : "12 Nov 2014 09:29:22 GMT"
> },
> "global" : {
> "internal.generated.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/generated",
> "application.name" : "test-agent-launchfail",
> "slider.cluster.directory.permissions" : "0770",
> "internal.provider.name" : "agent",
> "internal.data.dir.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/database",
> "internal.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "internal.snapshot.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/snapshot",
> "internal.chaos.monkey.interval.seconds" : "60",
> "slider.data.directory.permissions" : "0770",
> "internal.container.failure.shortlife" : "60000",
> "internal.chaos.monkey.enabled" : "true",
> "internal.am.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp/appmaster",
> "internal.container.failure.threshold" : "5"
> },
> "credentials" : { },
> "components" : { }
> },
> "resources": {
> "schema" : "http://example.org/specification/v2.0.0",
> "metadata" : { },
> "global" : { },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "yarn.memory" : "256",
> "yarn.vcores" : "1",
> "yarn.component.instances" : "1"
> },
> "COMMAND_LOGGER" : {
> "yarn.memory" : "128",
> "yarn.role.priority" : "1",
> "yarn.component.instances" : "0"
> }
> }
> },
> "appConf" :{
> "schema" : "http://example.org/specification/v2.0.0",
> "metadata" : { },
> "global" : {
> "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "jvm.heapsize" : "256M",
> "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> },
> "COMMAND_LOGGER" : {
> "site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
> "site.fs.default.name" : "hdfs://172.31.11.99:8020",
> "site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
> "site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
> "zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "java_home" : "/usr/jdk64/jdk1.7.0_67",
> "site.global.application_id" : "CommandLogger",
> "internal.chaos.monkey.probability.amlaunchfailure" : "10000",
> "site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
> "env.MALLOC_ARENA_MAX" : "4",
> "zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
> "internal.chaos.monkey.interval.seconds" : "60",
> "internal.chaos.monkey.enabled" : "true",
> "zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
> "site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
> "application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
> "site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
> "site.global.security_enabled" : "false"
> }
> }
> }}
> 2014-11-12 09:29:35,619 [main] INFO agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(133)) - Validating app definition .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
> 2014-11-12 09:29:35,620 [main] DEBUG state.AppState (AppState.java:updateResourceDefinitions(649)) - Updating resources to {
> "schema" : "http://example.org/specification/v2.0.0",
> "metadata" : { },
> "global" : { },
> "credentials" : { },
> "components" : {
> "slider-appmaster" : {
> "yarn.memory" : "256",
> "yarn.vcores" : "1",
> "yarn.component.instances" : "1"
> },
> "COMMAND_LOGGER" : {
> "yarn.memory" : "128",
> "yarn.role.priority" : "1",
> "yarn.component.instances" : "0"
> }
> }
> }
> 2014-11-12 09:29:35,704 [main] INFO state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
> 2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of slider-appmaster; was 0
> 2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of COMMAND_LOGGER; was 0
> 2014-11-12 09:29:35,704 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:reviewRequestAndReleaseNodes(1524)) - reviewRequestAndReleaseNodes(flexCluster)
> 2014-11-12 09:29:35,705 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,705 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:waitForAMCompletionSignal(1243)) - blocking until signalled to terminate
> 2014-11-12 09:29:35,705 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
> 2014-11-12 09:29:35,705 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1288)) - Triggering shutdown of the AM: stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
> 2014-11-12 09:29:35,706 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,706 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:stateChanged(1963)) - Process has exited with exit code 0 mapped to 0 -ignoring
> 2014-11-12 09:29:35,706 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:finish(1299)) - Stopped forked process: exit code=0
> 2014-11-12 09:29:35,707 [main] INFO workflow.WorkflowCompositeService (WorkflowCompositeService.java:stateChanged(123)) - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED
> 2014-11-12 09:29:35,707 [main] INFO state.AppState (AppState.java:releaseAllContainers(1843)) - Releasing 0 containers
> 2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1317)) - Application completed. Signalling finish to RM
> 2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1320)) - Unregistering AM status=FAILED message=Chaos monkey triggered launch failure
> 2014-11-12 09:29:35,716 [main] INFO impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(383)) - Waiting for application to be successfully unregistered.
> 2014-11-12 09:29:35,818 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:runService(529)) - Exiting AM; final exit code = 0
> 2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1559)) - in executeNodeReview(flexCluster)
> 2014-11-12 09:29:35,818 [main] DEBUG main.ServiceLauncher (ServiceLauncher.java:launchService(189)) - Service exited with exit code 0
> 2014-11-12 09:29:35,818 [AmExecutor-006] INFO appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1561)) - Ignoring node review operation: shutdown in progress
> 2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG state.AppState (AppState.java:reviewRequestAndReleaseNodes(1599)) - in reviewRequestAndReleaseNodes()
> 2014-11-12 09:29:35,818 [AmExecutor-006] INFO state.AppState (AppState.java:reviewOneRole(1684)) - Reviewing RoleStatus{name='COMMAND_LOGGER', key=1, minimum=0, maximum=1, desired=0, actual=0, requested=0, releasing=0, failed=0, started=0, startFailed=0, completed=0, failureMessage=''} : expected 0
> 2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG state.AppState (AppState.java:checkFailureThreshold(1620)) - Failure count of component: COMMAND_LOGGER: 0, threshold=5
> 2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
> 2014-11-12 09:29:35,821 [main] INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 0
> 2014-11-12 09:29:35,821 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook executing
> 2014-11-12 09:29:35,822 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:50007
> 2014-11-12 09:29:35,825 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:52395
> 2014-11-12 09:29:35,829 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook complete
> 2014-11-12 09:29:35,834 [Thread-1] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
> 2014-11-12 09:29:36,037 [Thread-1] INFO ipc.Server (Server.java:stop(2437)) - Stopping server on 40684
> 2014-11-12 09:29:36,039 [IPC Server listener on 40684] INFO ipc.Server (Server.java:run(706)) - Stopping IPC Server listener on 40684
> 2014-11-12 09:29:36,039 [IPC Server Responder] INFO ipc.Server (Server.java:run(832)) - Stopping IPC Server Responder
> 2014-11-12 09:29:36,040 [Thread-1] DEBUG actions.QueueService (QueueService.java:schedule(91)) - Scheduling org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
> 2014-11-12 09:29:36,040 [AMRM Callback Handler Thread] INFO impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(276)) - Interrupted while waiting for queue
> java.lang.InterruptedException
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
> 2014-11-12 09:29:36,042 [AmExecutor-005] DEBUG actions.QueueService (QueueService.java:run(176)) - Propagating org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
> 2014-11-12 09:29:36,042 [AmExecutor-005] INFO actions.QueueService (QueueService.java:run(179)) - QueueService processor terminated
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)