You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "Bobby Powers (JIRA)" <ji...@apache.org> on 2011/03/10 23:41:59 UTC
[jira] Created: (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp support
by doubling the max Buffer length
[PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
--------------------------------------------------------------------------
Key: SSHD-112
URL: https://issues.apache.org/jira/browse/SSHD-112
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Bobby Powers
Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
Not sure why it was 32k, but this causes sshfs on the mac (but not
linux, strangely) to break, as it tried to send 65535 bytes of data at a
time. Since it seems like an arbitrary limit I doubled it. The
relevant (I think) part of the spec is rfc4253 section 6.1:
All implementations MUST be able to process packets with an
uncompressed payload length of 32768 bytes or less and a total packet
size of 35000 bytes or less... Implementations SHOULD support longer
packets, where they might be needed.
Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.8
flags=0x00000000
max_readahead=0x00100000
failed to detect remote user ID
INIT: 7.8
flags=0x00000000
max_readahead=0x00100000
max_write=0x00010000
I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (SSHD-112) [PATCH][SFTP] fix mac
sshfs sftp support by doubling the max Buffer length
Posted by "Taras Puchko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029478#comment-13029478 ]
Taras Puchko edited comment on SSHD-112 at 5/5/11 6:16 PM:
-----------------------------------------------------------
This patch also fixes "IllegalStateException: Bad item length: 32829" when using KDE Dolphin.
was (Author: tarasp):
I have tested Mac OS X sshfs against trunk SSHD without this patch and the problem remains.
So, I think, this patch has nothing to do with the file corruption but it fixes "IllegalStateException: Bad item length: 32829" for me when using KDE.
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Bobby Powers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016047#comment-13016047 ]
Bobby Powers commented on SSHD-112:
-----------------------------------
Thanks Guillaume,
I don't think we should apply the patch yet. While it allows sshfs on the mac to connect, we end up with corrupted files if we try to copy a smaller file over an existing file. See:
http://groups.google.com/group/macfuse/browse_thread/thread/7bbc76f6a3e34e2f
I haven't had time to look at this in a week or so, but I _think_ whats happening is that the client (sshfs) issues a STAT call, thinks the file we're writing into is out of date, fetches the 'new' (existing) contents from the server, copies in the new content over the first X bytes of the previous file and then saves it (with the length of the previous file). Of note is that this doesn't happen when mac sshfs is connected to an openssh sftp implementation.
I think that failing to transfer files is better than silently corrupting them, so its better to hold off on merging this until we can work out what the STAT issue is. Any insights or hunches you might have are welcome :)
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Bobby Powers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011358#comment-13011358 ]
Bobby Powers commented on SSHD-112:
-----------------------------------
actually you probably don't want to merge this just yet :) We've found an issue uploading from the mac implementation of sshfs (the last 256 bytes of an xml file is repeated), and I can't be sure its not due to this. I'll try have an answer by the end of the weekend.
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Taras Puchko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029478#comment-13029478 ]
Taras Puchko commented on SSHD-112:
-----------------------------------
I have tested Mac OS X sshfs against trunk SSHD without this patch and the problem remains.
So, I think, this patch has nothing to do with the file corruption but it fixes "IllegalStateException: Bad item length: 32829" for me when using KDE.
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp support
by doubling the max Buffer length
Posted by "Bobby Powers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bobby Powers updated SSHD-112:
------------------------------
Attachment: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
fix
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Will Glass-Husain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011279#comment-13011279 ]
Will Glass-Husain commented on SSHD-112:
----------------------------------------
Just wanted to note that we are using this in production and it's solved some crazy problems with various clients. I do encourage the committers to add this to trunk.
(note - I'm a colleague of Bobby Powers).
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015851#comment-13015851 ]
Guillaume Nodet commented on SSHD-112:
--------------------------------------
You guys have any more informations ? I'm happy to commit the patch if that solves a problem.
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (SSHD-112) [PATCH][SFTP] fix mac sshfs sftp
support by doubling the max Buffer length
Posted by "Bobby Powers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SSHD-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006545#comment-13006545 ]
Bobby Powers commented on SSHD-112:
-----------------------------------
bump. any comments?
> [PATCH][SFTP] fix mac sshfs sftp support by doubling the max Buffer length
> --------------------------------------------------------------------------
>
> Key: SSHD-112
> URL: https://issues.apache.org/jira/browse/SSHD-112
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Bobby Powers
> Attachments: 0001-Buffer-increase-max-length-to-64-KiB-from-32-and-abs.patch
>
>
> Not sure why it was 32k, but this causes sshfs on the mac (but not
> linux, strangely) to break, as it tried to send 65535 bytes of data at a
> time. Since it seems like an arbitrary limit I doubled it. The
> relevant (I think) part of the spec is rfc4253 section 6.1:
> All implementations MUST be able to process packets with an
> uncompressed payload length of 32768 bytes or less and a total packet
> size of 35000 bytes or less... Implementations SHOULD support longer
> packets, where they might be needed.
> Without this we're unable to transfer large files in via sftp from the mac sshfs file system. sshfs's debug logging shows this:
> unique: 0, opcode: INIT (26), nodeid: 0, insize: 56
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> failed to detect remote user ID
> INIT: 7.8
> flags=0x00000000
> max_readahead=0x00100000
> max_write=0x00010000
> I don't know where it gets this max_write from, as far as I can tell max_write should be set in AbstractChannel.configureWindow to 32 KiB (line 187). I didn't see any window renegotiation in the logs, but it might have slipped by.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira