You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/10/07 16:21:00 UTC
[jira] [Work logged] (SSHD-1173) SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
[ https://issues.apache.org/jira/browse/SSHD-1173?focusedWorklogId=814732&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814732 ]
ASF GitHub Bot logged work on SSHD-1173:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Oct/22 16:20
Start Date: 07/Oct/22 16:20
Worklog Time Spent: 10m
Work Description: tomaswolf opened a new pull request, #251:
URL: https://github.com/apache/mina-sshd/pull/251
Remove the option not to send a chunk when the remote window is smaller than the packet size, and we're trying to write more than the remote widow size bytes. This option may lead to hangs with peers that wait with sending their window adjustment until the window size is very low, or even zero.
RFC 4254 does not mandate any particular point at which an SSH implementation should send a window adjustment. Implementations doing so only once the window has been fully consumed are well within the bounds of the specification.
Therefore remove this option altogether, and always send some data if the window size is > 0. Refactor the writePacket() method to make it much simpler and clearer.
Client options removed:
* SftpModuleProperties.CHUNK_IF_WINDOW_LESS_THAN_PACKET
* CoreModuleProperties.ASYNC_SERVER_STDOUT_CHUNK_BELOW_WINDOW_SIZE
* CoreModuleProperties.ASYNC_SERVER_STDERR_CHUNK_BELOW_WINDOW_SIZE
* ChannelAsyncOutputStream(Channel, byte, boolean) constructor
These options were by default all false, potentially leading to hangs as described above. Already [SSHD-1123] had pointed out that problem, and bug [SSHD-1207] might also be caused by this. The feature was introduced in [SSHD-979] in commit e85b67e0 and later modified in commit 536d0663.
Issue Time Tracking
-------------------
Worklog Id: (was: 814732)
Remaining Estimate: 0h
Time Spent: 10m
> SFTP worker threads got stuck while processing PUT methods against one specific SFTP server
> -------------------------------------------------------------------------------------------
>
> Key: SSHD-1173
> URL: https://issues.apache.org/jira/browse/SSHD-1173
> Project: MINA SSHD
> Issue Type: Question
> Affects Versions: 2.6.0
> Reporter: Pavel Pohner
> Priority: Critical
> Labels: SFTP, mina, sshd
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hey guys,
> Recently I've encountered one remote SFTP server, that causes SFTP worker threads to enter TIMED_WAITING state, from which they do not recover. Please, take a look into this thread dump:
> {code:java}
> SFTP-worker-3 #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s tid=0x00007fe41c005800 nid=0x18808 in Object.wait() [0x00007fe145b1d000]SFTP-worker-3" #2155 prio=5 os_prio=0 cpu=61692.09ms elapsed=1136151.86s tid=0x00007fe41c005800 nid=0x18808 in Object.wait() [0x00007fe145b1d000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting on <no object reference available> at org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69) - waiting to re-lock in wait() <0x00000006e3bbc420> (a org.apache.sshd.common.channel.IoWriteFutureImpl) at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:109) at org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:39) at org.apache.sshd.common.io.AbstractIoWriteFuture.verify(AbstractIoWriteFuture.java:30) at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43) at org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292) at org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210) at org.apache.sshd.sftp.client.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141) at java.io.InputStream.transferTo(java.base@11.0.10/InputStream.java:705) at com.mina.command.Put.replyInto(Put.java:55) at com.sftpserver.MinaSftpHandler.channelReadInternal(MinaSftpHandler.java:251) at com.sftpserver.MinaSftpHandler.lambda$channelRead$0(MinaSftpHandler.java:125) at com.sftpserver.MinaSftpHandler$$Lambda$392/0x0000000800764040.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.10/Executors.java:515) at java.util.concurrent.FutureTask.run(java.base@11.0.10/FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628) at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
> {code}
> Seems like the thread is waiting for lock in org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java), I'm currently not sure what thread holds the lock and why it's not ever released.
> Also, I'm not able to reproduce this, but it constantly happens to all the PUT methods targeting one specific remote server (which I don't own), so I suppose there would be something odd with the remote server, but it doesn't mean, that we shouldn't be able to deal with that.
> Have you ever seen such behavior? Could it be Mina sshd's bug? (seems like verify() in org.apache.sshd.sftp.client.impl.DefaultSftpClient.send(DefaultSftpClient.java:292) is called without any timeout, which then defaults to LONG.MAX here: at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:43), shouldn't we have configurable timeout here? or what's the point of default timeout? what are we really waiting for in at org.apache.sshd.common.future.DefaultSshFuture.await0(DefaultSshFuture.java:69)?)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org