You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Duncan Godwin (JIRA)" <ji...@apache.org> on 2016/08/04 15:21:20 UTC
[jira] [Resolved] (BROOKLYN-319) RabbitMQ restart failure
[ https://issues.apache.org/jira/browse/BROOKLYN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Duncan Godwin resolved BROOKLYN-319.
------------------------------------
Resolution: Fixed
> RabbitMQ restart failure
> ------------------------
>
> Key: BROOKLYN-319
> URL: https://issues.apache.org/jira/browse/BROOKLYN-319
> Project: Brooklyn
> Issue Type: Bug
> Environment: Centos 6 on BlueBox and Google Compute
> Reporter: Duncan Godwin
>
> RabbitMQ intermittently fails to come back up after restart of the entity on some clouds (BlueBox and Google Compute known to fail).
> The brooklyn.debug.log shows the following for stopping and starting:
> {code}
> 2016-07-28 09:49:58,719 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10422]: stopping RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)] (env {RABBITMQ_
> HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=567
> 2, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid, RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}): #!/bin/ba
> sh -e
> ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0" ; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46" ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl
> stop
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]: [d5kw04uk46@10.104.2.181:stdout] Stopping and halting node 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5' ...
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]: [d5kw04uk46@10.104.2.181:stdout] Executed /tmp/brooklyn-20160728-094958719-Sndu-stopping_RabbitBrokerImpl_id_d.sh, result 0
> 2016-07-28 09:49:59,072 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10422]: stopping RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)], completed: return status 0
> 2016-07-28 09:49:59,078 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10385]: launching RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)] (env {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid, RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}): #!/bin/bash -e
> ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0" ; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46" ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; nohup ./sbin/rabbitmq-server > console-out.log 2> console-err.log & ; for i in {1..60}
> do
> grep 'Starting broker... completed' console-out.log && exit
> sleep 1
> done ; echo "Couldn't determine if rabbitmq-server is running" ; exit 1
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]: [d5kw04uk46@10.104.2.181:stdout] Starting broker... completed with 0 plugins.
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]: [d5kw04uk46@10.104.2.181:stdout] Executed /tmp/brooklyn-20160728-094959078-mjxF-launching_RabbitBrokerImpl_id_.sh, result 0
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10385]: launching RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)], completed: return status 0
> {code}
> the console-out.log of the RabbitMQ instance contains:
> {code}
> ERROR: node with name "d5kw04uk46" already running on "qa-scheduler-rabbitmq-server-be5"
> {code}
> The brooklyn.debug.log then contains the following check running messages
> {code}
> 2016-07-28 14:05:12,642 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-12886]: check-running RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)] (env {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46, RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid, RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}): #!/bin/bash -e
> ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0" ; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46" ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl -q status
> 2016-07-28 14:05:12,982 DEBUG o.a.b.u.c.i.w.w.Winrm4jTool [brooklyn-execmanager-E26Om0Sg-12879]: Finished WinRM exec on Administrator@217.115.71.201:11049 done after 2.01s (connected in 0ms)
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350446]: [d5kw04uk46@10.104.2.181:stdout] Executed /tmp/brooklyn-20160728-140512642-NDCx-check-running_RabbitBrokerImpl.sh, result 69
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] Error: unable to connect to node 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5': nodedown
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] DIAGNOSTICS
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] ===========
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] attempted to contact: ['d5kw04uk46@qa-scheduler-rabbitmq-server-be5']
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] d5kw04uk46@qa-scheduler-rabbitmq-server-be5:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] * connected to epmd (port 4369) on qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] * epmd reports: node 'd5kw04uk46' not running at all
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] no other nodes on qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] * suggestion: start the node
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] current node details:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] - node name: 'rabbitmq-cli-23557@qa-scheduler-rabbitmq-server-be5'
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] - home dir: /home/users/amp
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr] - cookie hash: hGw+myYNiff8VA9lwwfh0g==
> {code}
> The RabbitMQ service is not runing on the machine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)