You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Aled Sage (JIRA)" <ji...@apache.org> on 2015/01/09 11:53:34 UTC
[jira] [Created] (BROOKLYN-115) ssh failure repeatedly (out of
retries: Timeout expired) on health-check after successfully provisoining
jboss in CentOS on vcloud-director
Aled Sage created BROOKLYN-115:
----------------------------------
Summary: ssh failure repeatedly (out of retries: Timeout expired) on health-check after successfully provisoining jboss in CentOS on vcloud-director
Key: BROOKLYN-115
URL: https://issues.apache.org/jira/browse/BROOKLYN-115
Project: Brooklyn
Issue Type: Bug
Affects Versions: 0.7.0-SNAPSHOT
Reporter: Aled Sage
I successfully provisioned a Jboss7 entity to a CentOS 6.4 VM in VMware's vcloud air (running vcloud-director 5.5, going over NAT).
However, several hours after the VM was provisioned it went on-fire because the check-running script (which goes over ssh to ensure the script is still running) began to fail repeatedly. Note this is using the new "async exec", but that is unconnected because it is *every* ssh command that fails rather than the long-poll.
{noformat}
Caused by: brooklyn.util.internal.ssh.SshException: (amp@23.92.230.21:11955) (amp@23.92.230.21:11955) error acquiring Shell(command=[[touch /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stdout /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stderr /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.exitstatus /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.pid, ( /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.sh > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stdout 2> /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stderr < /dev/null ; echo $? > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.exitstatus ) & disown, echo $! > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.pid, RESULT=$?, echo Executing async /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.sh, exit $RESULT]]) (attempt 1/1, in time 1m/2m); out of retries: Timeout expired
at brooklyn.util.internal.ssh.SshAbstractTool.propagate(SshAbstractTool.java:169) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:584) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:537) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool$3.run(SshjTool.java:354) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool.execScriptAsyncAndPoll(SshjTool.java:478) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool.execScript(SshjTool.java:320) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$1.exec(ExecWithLoggingHelpers.java:83) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:167) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:1) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.pool.BasicPool.exec(BasicPool.java:147) ~[brooklyn-utils-common-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.location.basic.SshMachineLocation.execSsh(SshMachineLocation.java:495) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
at brooklyn.location.basic.SshMachineLocation$11.execWithTool(SshMachineLocation.java:635) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.system.internal.ExecWithLoggingHelpers.execWithLogging(ExecWithLoggingHelpers.java:165) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.system.internal.ExecWithLoggingHelpers.execScript(ExecWithLoggingHelpers.java:81) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.location.basic.SshMachineLocation.execScript(SshMachineLocation.java:628) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
at brooklyn.entity.basic.AbstractSoftwareProcessSshDriver.execute(AbstractSoftwareProcessSshDriver.java:322) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.entity.basic.lifecycle.ScriptHelper.executeInternal(ScriptHelper.java:363) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
... 8 common frames omitted
Caused by: net.schmizz.sshj.connection.ConnectionException: Timeout expired
at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:32) ~[sshj-0.8.1.jar:na]
at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:26) ~[sshj-0.8.1.jar:na]
at net.schmizz.concurrent.Promise.retrieve(Promise.java:139) ~[sshj-0.8.1.jar:na]
at net.schmizz.concurrent.Event.await(Event.java:103) ~[sshj-0.8.1.jar:na]
at net.schmizz.sshj.connection.channel.AbstractChannel.join(AbstractChannel.java:282) ~[sshj-0.8.1.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:932) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:1) ~[patch-ssh-longpolling-retry.jar:na]
at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:551) ~[patch-ssh-longpolling-retry.jar:na]
... 23 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Timeout expired
... 29 common frames omitted
{noformat}
Trying to ssh manually to the VM (from the Brooklyn VM), I get:
{noformat}
[amp@AMP amp]$ ssh amp@23.92.230.21 -p 11955
Connection to 23.92.230.21 closed by remote host.
Connection to 23.92.230.21 closed.
{noformat}
>From that Brooklyn VM, I can successfully ssh to the box as a different user though (which uses password rather than ssh key).
>From my mac laptop, it is simiilar(ish). I can ssh as the different user, but when I try as the user amp I get a slightly different error:
{noformat}
Aleds-MacBook-Pro:vchs-ssh-hangs-20141222 aled $ssh -i ~/.ssh/id_rsa-canopy-tai-server amp@23.92.230.21 -p 11955
Write failed: Broken pipe
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)