You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "min yun law (Jira)" <ji...@apache.org> on 2020/12/16 17:49:00 UTC
[jira] [Commented] (SSHD-1078) SCP/SFTP failed for big file upload
[ https://issues.apache.org/jira/browse/SSHD-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250495#comment-17250495 ]
min yun law commented on SSHD-1078:
-----------------------------------
test using mina 2.6.0 with sftp with three big files uploading to the same host at the same time, and a 10 loops, look like it pass without issue. look like it fix the issue in 2.6.0
-rw-r--r--. 1 root root 5.2G Dec 16 12:34 p30032947_187000DBRU_Linux-x86-64.zip.sftp
-rw-r--r--. 1 root root 8.1G Dec 16 12:37 p30397820_188000GIRU_Linux-x86-64.zip.sftp
-rw-r--r--. 1 root root 8.1G Dec 16 12:37 p30032946_187000GIRU_Linux-x86-64.zip.sftp
retry count: 0
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 288s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 391s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 391s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 1
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 291s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 403s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 405s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 2
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 292s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 419s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 425s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 3
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 330s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 444s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 444s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 4
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 285s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 399s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 400s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 5
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 331s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 444s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 445s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 6
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 376s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 497s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 497s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
retry count: 7
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 286s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 408s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 409s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 8
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 298s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 409s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 412s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
retry count: 9
Using SFTP for upload....
Upload file p30032947_187000DBRU_Linux-x86-64.zip in 356s
*****done for p30032947_187000DBRU_Linux-x86-64.zip*************
Upload file p30397820_188000GIRU_Linux-x86-64.zip in 499s
*****done for p30397820_188000GIRU_Linux-x86-64.zip*************
Upload file p30032946_187000GIRU_Linux-x86-64.zip in 500s
*****done for p30032946_187000GIRU_Linux-x86-64.zip*************
> SCP/SFTP failed for big file upload
> -----------------------------------
>
> Key: SSHD-1078
> URL: https://issues.apache.org/jira/browse/SSHD-1078
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.5.1
> Reporter: min yun law
> Priority: Major
>
> Trying to merge product from jsch to mina sshd that keep the same SSH architecture (lets call SessionPool) which for each remote node, there is only one session object and limited channels (<=8) for executing commands and file operations. The SessionPool manages the session/channels for nodes and make sure only 8 threads(ssh channels) per node that can be run concurrently. This architecture works for Jsch for many years, but when using mina sshd, this architecture wont work when with upload big file, we always see following exception:
> {code:java}
> java.io.EOFException: Channel is being closed
> at org.apache.sshd.common.channel.AbstractChannel$2.<init>(AbstractChannel.java:794)
> at org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:792)
> at org.apache.sshd.common.channel.ChannelAsyncOutputStream.doWriteIfPossible(ChannelAsyncOutputStream.java:168)
> at org.apache.sshd.common.channel.ChannelAsyncOutputStream.writePacket(ChannelAsyncOutputStream.java:70)
> at org.apache.sshd.client.subsystem.sftp.impl.DefaultSftpClient.send(DefaultSftpClient.java:292)
> at org.apache.sshd.client.subsystem.sftp.impl.SftpOutputStreamAsync.flush(SftpOutputStreamAsync.java:210)
> at org.apache.sshd.client.subsystem.sftp.impl.SftpOutputStreamAsync.write(SftpOutputStreamAsync.java:141)
> {code}
> From SSH server log, we see follow error:
> {code:java}
> Aug 19 15:56:38 xxxx sftp-server[202217]: error: Unknown message 0
> Aug 19 15:56:38 xxxx sftp-server[202217]: error: msg_len 0 < consumed 1
> {code}
> other client side exceptions by mina sshd as:
> org.apache.sshd.common.channel.WindowClosedException: Already closed: Window[client/remote](ChannelExec[id=79, recipient=1]-ClientSessionImp
>
> ----
> After many tries, finally I found that for big file upload/download, I have to give its own SshClient/ClientSession objects and then there will no such channel closing exception during file upload.
> That means we cannot share the same SshClient and ClientSession object per remote node to execute commands and file operations, and there may has bug in ClientSession when handle channels in this situation.
> the reason that we like to share the same SshClient/ClientSession for the same remote node is to manage the session/channels and close them when no usage, also to reduce the cost to allocate those objects.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org