You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Aled Sage (JIRA)" <ji...@apache.org> on 2015/01/16 11:23:34 UTC
[jira] [Resolved] (BROOKLYN-115) ssh failure repeatedly (out of
retries: Timeout expired) on health-check after successfully provisoining
jboss in CentOS on vcloud-director
[ https://issues.apache.org/jira/browse/BROOKLYN-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aled Sage resolved BROOKLYN-115.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.7.0-SNAPSHOT
Fixed in https://github.com/apache/incubator-brooklyn/pull/448/files#diff-4 by ensuring those `tail` commands are terminated.
> ssh failure repeatedly (out of retries: Timeout expired) on health-check after successfully provisoining jboss in CentOS on vcloud-director
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BROOKLYN-115
> URL: https://issues.apache.org/jira/browse/BROOKLYN-115
> Project: Brooklyn
> Issue Type: Bug
> Affects Versions: 0.7.0-SNAPSHOT
> Reporter: Aled Sage
> Assignee: Aled Sage
> Fix For: 0.7.0-SNAPSHOT
>
>
> I successfully provisioned a Jboss7 entity to a CentOS 6.4 VM in VMware's vcloud air (running vcloud-director 5.5, going over NAT).
> However, several hours after the VM was provisioned it went on-fire because the check-running script (which goes over ssh to ensure the script is still running) began to fail repeatedly. Note this is using the new "async exec", but that is unconnected because it is *every* ssh command that fails rather than the long-poll.
> {noformat}
> Caused by: brooklyn.util.internal.ssh.SshException: (amp@23.92.230.21:11955) (amp@23.92.230.21:11955) error acquiring Shell(command=[[touch /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stdout /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stderr /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.exitstatus /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.pid, ( /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.sh > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stdout 2> /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.stderr < /dev/null ; echo $? > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.exitstatus ) & disown, echo $! > /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.pid, RESULT=$?, echo Executing async /tmp/brooklyn-20150109-063334594-LtUy-check-running_JBoss7ServerImpl.sh, exit $RESULT]]) (attempt 1/1, in time 1m/2m); out of retries: Timeout expired
> at brooklyn.util.internal.ssh.SshAbstractTool.propagate(SshAbstractTool.java:169) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:584) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:537) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool$3.run(SshjTool.java:354) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool.execScriptAsyncAndPoll(SshjTool.java:478) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool.execScript(SshjTool.java:320) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$1.exec(ExecWithLoggingHelpers.java:83) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:167) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.task.system.internal.ExecWithLoggingHelpers$3.apply(ExecWithLoggingHelpers.java:1) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.pool.BasicPool.exec(BasicPool.java:147) ~[brooklyn-utils-common-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.location.basic.SshMachineLocation.execSsh(SshMachineLocation.java:495) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
> at brooklyn.location.basic.SshMachineLocation$11.execWithTool(SshMachineLocation.java:635) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.task.system.internal.ExecWithLoggingHelpers.execWithLogging(ExecWithLoggingHelpers.java:165) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.util.task.system.internal.ExecWithLoggingHelpers.execScript(ExecWithLoggingHelpers.java:81) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.location.basic.SshMachineLocation.execScript(SshMachineLocation.java:628) ~[patch-ssh-longpolling-retry.jar:0.7.0-SNAPSHOT]
> at brooklyn.entity.basic.AbstractSoftwareProcessSshDriver.execute(AbstractSoftwareProcessSshDriver.java:322) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at brooklyn.entity.basic.lifecycle.ScriptHelper.executeInternal(ScriptHelper.java:363) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> ... 8 common frames omitted
> Caused by: net.schmizz.sshj.connection.ConnectionException: Timeout expired
> at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:32) ~[sshj-0.8.1.jar:na]
> at net.schmizz.sshj.connection.ConnectionException$1.chain(ConnectionException.java:26) ~[sshj-0.8.1.jar:na]
> at net.schmizz.concurrent.Promise.retrieve(Promise.java:139) ~[sshj-0.8.1.jar:na]
> at net.schmizz.concurrent.Event.await(Event.java:103) ~[sshj-0.8.1.jar:na]
> at net.schmizz.sshj.connection.channel.AbstractChannel.join(AbstractChannel.java:282) ~[sshj-0.8.1.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:932) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool$ShellAction.create(SshjTool.java:1) ~[patch-ssh-longpolling-retry.jar:na]
> at brooklyn.util.internal.ssh.sshj.SshjTool.acquire(SshjTool.java:551) ~[patch-ssh-longpolling-retry.jar:na]
> ... 23 common frames omitted
> Caused by: java.util.concurrent.TimeoutException: Timeout expired
> ... 29 common frames omitted
> {noformat}
> Trying to ssh manually to the VM (from the Brooklyn VM), I get:
> {noformat}
> [amp@AMP amp]$ ssh amp@23.92.230.21 -p 11955
> Connection to 23.92.230.21 closed by remote host.
> Connection to 23.92.230.21 closed.
> {noformat}
> From that Brooklyn VM, I can successfully ssh to the box as a different user though (which uses password rather than ssh key).
> From my mac laptop, it is simiilar(ish). I can ssh as the different user, but when I try as the user amp I get a slightly different error:
> {noformat}
> Aleds-MacBook-Pro:vchs-ssh-hangs-20141222 aled $ssh -i ~/.ssh/id_rsa-canopy-tai-server amp@23.92.230.21 -p 11955
> Write failed: Broken pipe
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)