You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "smitsjelle (via GitHub)" <gi...@apache.org> on 2025/03/10 13:48:04 UTC
[I] SFTP writes with 'append' mode may insert instead [mina-sshd]
smitsjelle opened a new issue, #690:
URL: https://github.com/apache/mina-sshd/issues/690
### Version
2.14.0
### Bug description
I'm using Spring Integration to write files to an SFTP server. I noticed that on one specific server I get odd behavior. Instead of adding the contents to the existing file, it overrides the content. It seems to be inserting the data from position `0` until it is done. If there was already more data in the file, that data remains.
For example, if I have a `test.txt` on my server with content `abcdefghi` and I send `123456` with setting APPEND, the resulting file content is `123456ghi` instead of `abcdefghi123456`.
In order to do the append, Spring Integrations' [SFTPSession#append(..)](https://github.com/spring-projects/spring-integration/blob/be8c1e281e551af4645e3f7ce654474936985f11/spring-integration-sftp/src/main/java/org/springframework/integration/sftp/session/SftpSession.java#L144) method uses [AbstractSftpClient#write(String path, int bufferSize, Collection<OpenMode> mode)](https://github.com/apache/mina-sshd/blob/463c51bdc9cc8b124cb966b114a438ed255ee1de/sshd-sftp/src/main/java/org/apache/sshd/sftp/client/impl/AbstractSftpClient.java#L1241) to do appends. When I change Spring's method to use the [SftpClient#write(Handle handle, long fileOffset, byte[] src)](https://github.com/apache/mina-sshd/blob/463c51bdc9cc8b124cb966b114a438ed255ee1de/sshd-sftp/src/main/java/org/apache/sshd/sftp/client/SftpClient.java#L681C18-L681C67) instead, where I pass the current length of the file as its offset (retrieved by calling `sftpClient.stat(destination).getSize()`), the append works as expected.
It feels like the write method of `AbstractSftpClient` doesn't handle the append correctly if the supplied `OpenMode`s are `Write, Append, Create`. Is this correct, or is there something wrong with the parameters that are passed?
### Actual behavior
The data gets inserted from position 0 instead of getting added after the last position in the document. I.e.: when the inserted data is less than the length of the existing document, it gets overriden until the new data ends, and remains the same after that.
### Expected behavior
When the `OpenMode` 'Append' is provided, data is always added at the end of the file.
### Relevant log output
```Shell
```
### Other information
The 'getVersion' command for the server I'm encountering the issue for returns version 3. In testcases where I cannot reproduce the issue, the version is 6. Unfortunately, I don't have any further details on the configuration of the SFTP server I'm encountering the issue on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2713991881
As far as I can determine it looks like this is some GlobalScape SFTP server, so most likely running on Windows. No idea what version of OpenSSH they might be using, or how old that software reporting this version is.
SFTP can be run over OpenSSH as an external process, so even if this server used OpenSSH 7.9 it might still use a proprietary SFTP implementation. It also appears that on these servers, there is a separate permission setting to allow appending to files, but I have not found any information about what happens if a user doesn't have that permission and tries appending all the same. (Does one get a failure response? Or does it silently ignore the append flag?)
To me this very much looks like a server-side problem. Does appending to a remote file on a plain vanilla OpenSSH server work for you? (OpenSSH is always be SFTP v3.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "smitsjelle (via GitHub)" <gi...@apache.org>.
smitsjelle commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2718162861
Excuse me, missed that part.
Offset is the size, as fetched from the Attributes
```
SftpClient.Attributes attrs = this.sftpClient.stat(destination);
long offset = (attrs != null) ? attrs.getSize() : 0;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "smitsjelle (via GitHub)" <gi...@apache.org>.
smitsjelle commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2711062636
The 'doReadIdentification' reports 'Server version string: SSH-2.0-7.9.0.0_openssh'. Looks to me like a standard version, although it indeed is an older one. Yet I'd expect this version to the support for append protocol, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2718258567
Well, then it's clear why that worked: you're not making use of the append flag at all. I suspect this would also work if you didn't pass `SftpClient.OpenMode.Append`. You're writing at the current end of the file, and that's it.
So far this is consistent with having a "buggy server that ignores the append flag".
You mentioned that with JSch append with that server worked. As far as I see JSch gets the remote file size, and then explicitly sends SSH_FXF_WRITE requests with that as base offset.
Apache MINA sshd doesn't do this. SFTP v3 says that the append flag "forces all writes to append data at the end of the file", so the offset should just be ignored. Later drafts for SFTP v4/5/6 are more explicit and state "Data is always written at the end of the file. The offset field of SSH_FXP_WRITE requests is ignored."
So: still a buggy server IMO, but there is something we can do to try to work around this server issue. (At the cost of an additional `stat` remote call.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2718085061
What is `offset` in this snippet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2710820263
So you have an SFTP v3 server that ignores the SSH_FXF_APPEND flag given in the SSH_FXP_OPEN message and that instead always writes at the offset given in the SSH_FXP_WRITE messages. What kind of server is that? Maybe the identification string sent when the connection is established shows something interesting. (Visible with debug logging in the client.)
Not sure we can do anything about it in the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "smitsjelle (via GitHub)" <gi...@apache.org>.
smitsjelle commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2717733324
I did a quick test with some OpenSSH SFTP server (from this Docker image) that reports doReadIdentification `SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3`. For that one it does indeed work fine, which suggests that the issue might indeed be either server-specific or maybe somewhere in OpenSSH 7.9.
The reason why I'm not sure whether it is a server specific issue yet, is that this server was handling my append operations just fine when I used an older Spring Integration version that still used Jsch as SFTP library. Also, with a small change in the append code I seem to be able to do the append (see the code snippet below). Do you know if that is because it inherently does a _write from_ rather than a more dynamic append, or would this still be considered a 'normal' append?
This snippet does produce working append behavior for this server:
```
SftpClient.CloseableHandle handle = this.sftpClient.open(
destination,
SftpClient.OpenMode.Create,
SftpClient.OpenMode.Write,
SftpClient.OpenMode.Append);
this.sftpClient.write(handle, offset, inputStream.readAllBytes());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2711350025
I don't know. I have no idea what version "7.9.0.0_openssh" is. Not sure that would be OpenSSH 7.9. I find _four_ digits a bit strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #690:
URL: https://github.com/apache/mina-sshd/issues/690#issuecomment-2710958302
For instance OpenSSH can append only since version 6.3 (released 2013-09-13). Older OpenSSH versions would show the behavior you mentioned. Compare also https://github.com/hierynomus/sshj/issues/390#issuecomment-356213873 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
Re: [I] SFTP writes with 'append' mode may insert instead [mina-sshd]
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf closed issue #690: SFTP writes with 'append' mode may insert instead
URL: https://github.com/apache/mina-sshd/issues/690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org