[jira] [Created] (JCLOUDS-600) runScriptOnNode's sftp put can hang

["Andrew Bayer (JIRA)" <ji...@apache.org>]

Andrew Bayer created JCLOUDS-600:
------------------------------------

             Summary: runScriptOnNode's sftp put can hang
                 Key: JCLOUDS-600
                 URL: https://issues.apache.org/jira/browse/JCLOUDS-600
             Project: jclouds
          Issue Type: Bug
          Components: jclouds-compute
    Affects Versions: 1.7.3
            Reporter: Andrew Bayer


So I'm still digging at this one, but wanted to open a JIRA while I was still fresh on it. I'm seeing computeService.runScriptOnNode(...) hanging on the sftp phase a decent chunk of the time, and it's...annoying. I can tell when this has happened because sftp-server is still running on the VM, but the file is up and nothing progresses from there. I'm seeing the thread that's doing the runScriptOnNode looking like this:

{code}
Thread 6343: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(int, long) @bci=106, line=1037 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(int, long) @bci=25, line=1326 (Interpreted frame)
 - com.google.common.util.concurrent.AbstractFuture$Sync.get(long) @bci=3, line=268 (Interpreted frame)
 - com.google.common.util.concurrent.AbstractFuture.get(long, java.util.concurrent.TimeUnit) @bci=9, line=96 (Interpreted frame)
 - org.jclouds.compute.callables.BlockUntilInitScriptStatusIsZeroThenReturnOutput.get(long, java.util.concurrent.TimeUnit) @bci=3, line=194 (Interpreted frame)
 - org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSshAndBlockUntilComplete.doCall() @bci=14, line=60 (Interpreted frame)
 - org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call() @bci=27, line=77 (Interpreted frame)
 - org.jclouds.compute.internal.BaseComputeService.runScriptOnNode(java.lang.String, org.jclouds.scriptbuilder.domain.Statement, org.jclouds.compute.options.RunScriptOptions) @bci\
=109, line=615 (Interpreted frame)
 - org.jclouds.compute.internal.BaseComputeService.runScriptOnNode(java.lang.String, org.jclouds.scriptbuilder.domain.Statement) @bci=6, line=599 (Interpreted frame)
 - org.jclouds.compute.ComputeService$runScriptOnNode.call(java.lang.Object, java.lang.Object, java.lang.Object) @bci=22 (Interpreted frame)
{code}

And then additional threads like this:

{code}
Thread 6418: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
 - net.schmizz.sshj.connection.channel.ChannelInputStream.read(byte[], int, int) @bci=49, line=128 (Interpreted frame)
 - net.schmizz.sshj.sftp.PacketReader.readIntoBuffer(byte[], int, int) @bci=25, line=49 (Interpreted frame)
 - net.schmizz.sshj.sftp.PacketReader.getPacketLength() @bci=11, line=57 (Interpreted frame)
 - net.schmizz.sshj.sftp.PacketReader.readPacket() @bci=1, line=73 (Interpreted frame)
 - net.schmizz.sshj.sftp.PacketReader.run() @bci=8, line=85 (Interpreted frame)


Thread 6417: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=146 (Compiled frame)
 - net.schmizz.sshj.transport.Reader.run() @bci=41, line=68 (Compiled frame)
{code}

I'm not sure exactly how these threads are related (and these particular threads may not actually be directly related at all - there are a bunch of each sort of thread going, since I'm working across multiple instances, but there's one of each for each hung sftp, so far as I can tell).

I'm timing out the runScriptOnNode calls and then retrying them, and 90+% of the time, it seems to all work fine the second run through, so I'm really not sure what's going on. Anyone have any ideas?



--
This message was sent by Atlassian JIRA
(v6.2#6252)