You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by "Gour Saha (JIRA)" <ji...@apache.org> on 2014/11/13 02:13:23 UTC
[jira] [Created] (SLIDER-646) AgentLaunchFailureIT test failing at
times due to randomness of initial delay in Chaos Monkey
Gour Saha created SLIDER-646:
--------------------------------
Summary: AgentLaunchFailureIT test failing at times due to randomness of initial delay in Chaos Monkey
Key: SLIDER-646
URL: https://issues.apache.org/jira/browse/SLIDER-646
Project: Slider
Issue Type: Bug
Reporter: Gour Saha
Chaos Monkey initial delay should be deterministic. It is currently set to 60 seconds. Subsequent interval is also set to 60 secs. However AgentLaunchFailureIT fails at times because the AM does not get sufficient time to startup. In one failure scenario it has been seen to fail within 300 ms of Chaos Monkey setup. This test fails about once in every 10 attempts.
Here is the test output -
{code}
------------------------------------------------------------------------------
Test set: org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 43.503 sec <<< FAILURE! - in org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT
testAgentLaunchFailure(org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT) Time elapsed: 40.903 sec <<< FAILURE!
java.lang.AssertionError: Application Launch Failure, exit code 68
Chaos monkey triggered launch failure
at org.junit.Assert.fail(Assert.java:88)
at org.apache.slider.funtest.framework.CommandTestBase.createTemplatedSliderApplication(CommandTestBase.groovy:676)
at org.apache.slider.funtest.lifecycle.AgentLaunchFailureIT.testAgentLaunchFailure(AgentLaunchFailureIT.groovy:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
Here is the AM log snippet -
{code}
2014-11-12 09:29:34,989 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(764)) - Token YARN_AM_RM_TOKEN
2014-11-12 09:29:34,990 [main] INFO agent.AgentUtils (AgentUtils.java:getApplicationMetainfo(43)) - Reading metainfo at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,014 [main] INFO tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading metainfo.xml of size 1995
2014-11-12 09:29:35,096 [main] INFO agent.AgentUtils (AgentUtils.java:getDefaultConfig(64)) - Reading default config file configuration/cl-site.xml at .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,102 [main] INFO tools.SliderUtils (SliderUtils.java:getApplicationResourceInputStream(1692)) - Reading configuration/cl-site.xml of size 1270
2014-11-12 09:29:35,106 [main] INFO agent.HeartbeatMonitor (HeartbeatMonitor.java:start(46)) - Starting heartbeat monitor with interval 60000
2014-11-12 09:29:35,107 [Thread-36] DEBUG agent.HeartbeatMonitor (HeartbeatMonitor.java:run(65)) - Putting monitor to sleep for 60000 milliseconds
2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:buildInstance(502)) - Adding role COMMAND_LOGGER
2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:createDynamicProviderRole(585)) - Role COMMAND_LOGGER assigned priority 1
2014-11-12 09:29:35,181 [main] INFO state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
2014-11-12 09:29:35,253 [main] DEBUG state.RoleHistory (RoleHistory.java:onBootstrap(370)) - Role history bootstrapped
2014-11-12 09:29:35,268 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2183)) - Adding Chaos Monkey scheduled every 60 seconds (0 hours -delay 60
2014-11-12 09:29:35,269 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2195)) - Chaos Monkey has triggered AM Launch failure
2014-11-12 09:29:35,269 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action AM killer not enabled
2014-11-12 09:29:35,270 [main] DEBUG monkey.ChaosMonkeyService (ChaosMonkeyService.java:addTarget(66)) - Action Container killer not enabled
2014-11-12 09:29:35,270 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:maybeStartMonkey(2222)) - Chaos monkey not started
2014-11-12 09:29:35,271 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(879)) - HADOOP_USER_NAME='yarn'
2014-11-12 09:29:35,287 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(882)) - Registry service username =yarn
2014-11-12 09:29:35,308 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/yarn-site.xml
2014-11-12 09:29:35,342 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/core-site.xml
2014-11-12 09:29:35,375 [main] DEBUG tools.ConfigHelper (ConfigHelper.java:loadFromResource(511)) - loaded resources from file:/etc/hadoop/conf/hdfs-site.xml
2014-11-12 09:29:35,437 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1083)) - Service Record
ServiceRecord{description='Slider Application Master'; external endpoints: {{
"api" : "http://",
"addressType" : "uri",
"protocolType" : "webui",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091"
} ]
}; {
"api" : "classpath:org.apache.slider.management",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
} ]
}; {
"api" : "classpath:org.apache.slider.registry",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher.configurations",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher.exports",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
} ]
}; }; internal endpoints: {{
"api" : "classpath:org.apache.slider.agents.secure",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
} ]
}; {
"api" : "classpath:org.apache.slider.agents.oneway",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
} ]
}; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
2014-11-12 09:29:35,473 [main] INFO zk.RegistryOperationsService (RegistryOperationsService.java:bind(110)) - Bound at /users/yarn/services/org-apache-slider/test-agent-launchfail : ServiceRecord{description='Slider Application Master'; external endpoints: {{
"api" : "http://",
"addressType" : "uri",
"protocolType" : "webui",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091"
} ]
}; {
"api" : "classpath:org.apache.slider.management",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/mgmt"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher"
} ]
}; {
"api" : "classpath:org.apache.slider.registry",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/registry"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher.configurations",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/slider"
} ]
}; {
"api" : "classpath:org.apache.slider.publisher.exports",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "http://172.31.11.97:37091/ws/v1/slider/publisher/exports"
} ]
}; }; internal endpoints: {{
"api" : "classpath:org.apache.slider.agents.secure",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "https://172.31.11.97:50007/ws/v1/slider/agents"
} ]
}; {
"api" : "classpath:org.apache.slider.agents.oneway",
"addressType" : "uri",
"protocolType" : "REST",
"addresses" : [ {
"uri" : "https://172.31.11.97:52395/ws/v1/slider/agents"
} ]
}; }, attributes: {"yarn:persistence"="application" "yarn:id"="application_1415782602687_0008" }}
2014-11-12 09:29:35,535 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:registerServiceInstance(1085)) - Registered service under /users/yarn/services/org-apache-slider/test-agent-launchfail; absolute path /registry/users/yarn/services/org-apache-slider/test-agent-launchfail
2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(888)) - RM Webapp address 172.31.11.99:8088
2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(889)) - slider Webapp address http://172.31.11.97:37091
2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:createAndRunCluster(892)) - Application Master Initialization Completed
2014-11-12 09:29:35,543 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:startQueueProcessing(463)) - Queue Processing started
2014-11-12 09:29:35,544 [AmExecutor-005] INFO actions.QueueService (QueueService.java:run(171)) - QueueService processor started
2014-11-12 09:29:35,545 [AmExecutor-006] INFO actions.QueueExecutor (QueueExecutor.java:run(68)) - Queue Executor run() started
2014-11-12 09:29:35,545 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,545 [AmExecutor-006] INFO appmaster.SliderAppMaster (ActionStopSlider.java:execute(118)) - SliderAppMasterApi.stopCluster: Chaos monkey triggered launch failure
2014-11-12 09:29:35,619 [main] DEBUG agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(117)) - Validating conf {,
"internal": {
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : {
"create.hadoop.deployed.info" : "(no branch) @ae493e14a5f8a78bd6227e6d377bbef6",
"create.application.build.info" : "Slider Core-0.60.0.2.2.0.0-1947 Built against commit# ${buildNumber} on Java 1.7.0_67 by yarn",
"create.hadoop.build.info" : "2.6.0.2.2.0.0-1947",
"create.time.millis" : "1415784562671",
"create.time" : "12 Nov 2014 09:29:22 GMT"
},
"global" : {
"internal.generated.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/generated",
"application.name" : "test-agent-launchfail",
"slider.cluster.directory.permissions" : "0770",
"internal.provider.name" : "agent",
"internal.data.dir.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/database",
"internal.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp",
"internal.chaos.monkey.probability.amlaunchfailure" : "10000",
"internal.snapshot.conf.path" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/snapshot",
"internal.chaos.monkey.interval.seconds" : "60",
"slider.data.directory.permissions" : "0770",
"internal.container.failure.shortlife" : "60000",
"internal.chaos.monkey.enabled" : "true",
"internal.am.tmp.dir" : "hdfs://172.31.11.99:8020/user/yarn/.slider/cluster/test-agent-launchfail/tmp/appmaster",
"internal.container.failure.threshold" : "5"
},
"credentials" : { },
"components" : { }
},
"resources": {
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : { },
"global" : { },
"credentials" : { },
"components" : {
"slider-appmaster" : {
"yarn.memory" : "256",
"yarn.vcores" : "1",
"yarn.component.instances" : "1"
},
"COMMAND_LOGGER" : {
"yarn.memory" : "128",
"yarn.role.priority" : "1",
"yarn.component.instances" : "0"
}
}
},
"appConf" :{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : { },
"global" : {
"site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
"site.fs.default.name" : "hdfs://172.31.11.99:8020",
"site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
"site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
"zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
"java_home" : "/usr/jdk64/jdk1.7.0_67",
"site.global.application_id" : "CommandLogger",
"internal.chaos.monkey.probability.amlaunchfailure" : "10000",
"site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
"env.MALLOC_ARENA_MAX" : "4",
"zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
"internal.chaos.monkey.interval.seconds" : "60",
"internal.chaos.monkey.enabled" : "true",
"zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
"site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
"application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
"site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
"site.global.security_enabled" : "false"
},
"credentials" : { },
"components" : {
"slider-appmaster" : {
"jvm.heapsize" : "256M",
"site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
"site.fs.default.name" : "hdfs://172.31.11.99:8020",
"site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
"site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
"zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
"java_home" : "/usr/jdk64/jdk1.7.0_67",
"site.global.application_id" : "CommandLogger",
"internal.chaos.monkey.probability.amlaunchfailure" : "10000",
"site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
"env.MALLOC_ARENA_MAX" : "4",
"zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
"internal.chaos.monkey.interval.seconds" : "60",
"internal.chaos.monkey.enabled" : "true",
"zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
"site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
"application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
"site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
"site.global.security_enabled" : "false"
},
"COMMAND_LOGGER" : {
"site.dfs.namenode.kerberos.principal" : "nn/_HOST@EXAMPLE.COM",
"site.fs.default.name" : "hdfs://172.31.11.99:8020",
"site.cl-site.pattern.for.test.to.verify" : "verify this pattern",
"site.cl-site.logfile.location" : "${AGENT_LOG_ROOT}/operations.log",
"zookeeper.hosts" : "172.31.11.100,172.31.11.96,172.31.11.97",
"java_home" : "/usr/jdk64/jdk1.7.0_67",
"site.global.application_id" : "CommandLogger",
"internal.chaos.monkey.probability.amlaunchfailure" : "10000",
"site.fs.defaultFS" : "hdfs://172.31.11.99:8020",
"env.MALLOC_ARENA_MAX" : "4",
"zookeeper.path" : "/services/slider/users/yarn/test-agent-launchfail",
"internal.chaos.monkey.interval.seconds" : "60",
"internal.chaos.monkey.enabled" : "true",
"zookeeper.quorum" : "172.31.11.100,172.31.11.96,172.31.11.97",
"site.global.app_root" : "${AGENT_WORK_ROOT}/app/install/command-logger",
"application.def" : ".slider/package/CMD_LOGGER/apache-slider-command-logger.zip",
"site.cl-site.datetime.format" : "%A, %d. %B %Y %I:%M%p",
"site.global.security_enabled" : "false"
}
}
}}
2014-11-12 09:29:35,619 [main] INFO agent.AgentClientProvider (AgentClientProvider.java:validateInstanceDefinition(133)) - Validating app definition .slider/package/CMD_LOGGER/apache-slider-command-logger.zip
2014-11-12 09:29:35,620 [main] DEBUG state.AppState (AppState.java:updateResourceDefinitions(649)) - Updating resources to {
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : { },
"global" : { },
"credentials" : { },
"components" : {
"slider-appmaster" : {
"yarn.memory" : "256",
"yarn.vcores" : "1",
"yarn.component.instances" : "1"
},
"COMMAND_LOGGER" : {
"yarn.memory" : "128",
"yarn.role.priority" : "1",
"yarn.component.instances" : "0"
}
}
}
2014-11-12 09:29:35,704 [main] INFO state.AppState (AppState.java:buildRoleRequirementsFromResources(687)) - Role COMMAND_LOGGER has 0 instances specified
2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of slider-appmaster; was 0
2014-11-12 09:29:35,704 [main] DEBUG state.AppState (AppState.java:resetFailureCounts(1657)) - Resetting failure count of COMMAND_LOGGER; was 0
2014-11-12 09:29:35,704 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:reviewRequestAndReleaseNodes(1524)) - reviewRequestAndReleaseNodes(flexCluster)
2014-11-12 09:29:35,705 [main] DEBUG actions.QueueService (QueueService.java:put(85)) - Queueing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,705 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:waitForAMCompletionSignal(1243)) - blocking until signalled to terminate
2014-11-12 09:29:35,705 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,705 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1288)) - Triggering shutdown of the AM: stop: exit code = -1, FAILED: Chaos monkey triggered launch failure;
2014-11-12 09:29:35,706 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(71)) - Executing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,706 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:stateChanged(1963)) - Process has exited with exit code 0 mapped to 0 -ignoring
2014-11-12 09:29:35,706 [main] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:finish(1299)) - Stopped forked process: exit code=0
2014-11-12 09:29:35,707 [main] INFO workflow.WorkflowCompositeService (WorkflowCompositeService.java:stateChanged(123)) - Child service completed Service RoleLaunchService in state RoleLaunchService: STOPPED
2014-11-12 09:29:35,707 [main] INFO state.AppState (AppState.java:releaseAllContainers(1843)) - Releasing 0 containers
2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1317)) - Application completed. Signalling finish to RM
2014-11-12 09:29:35,707 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:finish(1320)) - Unregistering AM status=FAILED message=Chaos monkey triggered launch failure
2014-11-12 09:29:35,716 [main] INFO impl.AMRMClientImpl (AMRMClientImpl.java:unregisterApplicationMaster(383)) - Waiting for application to be successfully unregistered.
2014-11-12 09:29:35,818 [main] INFO appmaster.SliderAppMaster (SliderAppMaster.java:runService(529)) - Exiting AM; final exit code = 0
2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1559)) - in executeNodeReview(flexCluster)
2014-11-12 09:29:35,818 [main] DEBUG main.ServiceLauncher (ServiceLauncher.java:launchService(189)) - Service exited with exit code 0
2014-11-12 09:29:35,818 [AmExecutor-006] INFO appmaster.SliderAppMaster (SliderAppMaster.java:executeNodeReview(1561)) - Ignoring node review operation: shutdown in progress
2014-11-12 09:29:35,818 [AmExecutor-006] DEBUG state.AppState (AppState.java:reviewRequestAndReleaseNodes(1599)) - in reviewRequestAndReleaseNodes()
2014-11-12 09:29:35,818 [AmExecutor-006] INFO state.AppState (AppState.java:reviewOneRole(1684)) - Reviewing RoleStatus{name='COMMAND_LOGGER', key=1, minimum=0, maximum=1, desired=0, actual=0, requested=0, releasing=0, failed=0, started=0, startFailed=0, completed=0, failureMessage=''} : expected 0
2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG state.AppState (AppState.java:checkFailureThreshold(1620)) - Failure count of component: COMMAND_LOGGER: 0, threshold=5
2014-11-12 09:29:35,819 [AmExecutor-006] DEBUG actions.QueueExecutor (QueueExecutor.java:run(74)) - Completed org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@27f143e9 name='flexCluster', delay=0, attrs=4, sequenceNumber=2}
2014-11-12 09:29:35,821 [main] INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 0
2014-11-12 09:29:35,821 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook executing
2014-11-12 09:29:35,822 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:50007
2014-11-12 09:29:35,825 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSelectChannelConnector@0.0.0.0:52395
2014-11-12 09:29:35,829 [Shutdown] INFO mortbay.log (Slf4jLog.java:info(67)) - Shutdown hook complete
2014-11-12 09:29:35,834 [Thread-1] INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2014-11-12 09:29:36,037 [Thread-1] INFO ipc.Server (Server.java:stop(2437)) - Stopping server on 40684
2014-11-12 09:29:36,039 [IPC Server listener on 40684] INFO ipc.Server (Server.java:run(706)) - Stopping IPC Server listener on 40684
2014-11-12 09:29:36,039 [IPC Server Responder] INFO ipc.Server (Server.java:run(832)) - Stopping IPC Server Responder
2014-11-12 09:29:36,040 [Thread-1] DEBUG actions.QueueService (QueueService.java:schedule(91)) - Scheduling org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
2014-11-12 09:29:36,040 [AMRM Callback Handler Thread] INFO impl.AMRMClientAsyncImpl (AMRMClientAsyncImpl.java:run(276)) - Interrupted while waiting for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2014-11-12 09:29:36,042 [AmExecutor-005] DEBUG actions.QueueService (QueueService.java:run(176)) - Propagating org.apache.slider.server.appmaster.actions.ActionStopQueue@9731632 name='serviceStop: Service Action Queue in state Action Queue: STOPPED', delay=0, attrs=0, sequenceNumber=3}
2014-11-12 09:29:36,042 [AmExecutor-005] INFO actions.QueueService (QueueService.java:run(179)) - QueueService processor terminated
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)