[jira] [Comment Edited] (MINIFICPP-542) Add PutSFTP Support

["Daniel Bakai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MINIFICPP-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16828934#comment-16828934 ] 

Daniel Bakai edited comment on MINIFICPP-542 at 4/29/19 7:47 AM:
-----------------------------------------------------------------

[~phrocker] 

I’ve looked at curl an other possible implementations (libssh, libssh2, OpenSSH). curl itself uses libssh2 internally. I’ve found that while we could probably provide most of the things we need using just curl, we wouldn’t have the control I would be comfortable with. curl works in a “one download or upload operation on a file or directory, and random commands before and after that” model: you specify one upload or download, and other commands to execute before that (CURLOPT_QUOTE) and after that (CURLOPT_POSTQUOTE). However, you don’t really have much control over these commands. You can either flag them as fatal, if failed, or non-fatal, if failed, but after executing the whole operation, you don’t know the result for each of these commands, and if one of them fails, you don’t know which one did. You also don’t have the range of parameterization available for commands which are available natively in the protocol and also provided by libssh2, and some of which I see would be useful in implementing the processors. Because we need to execute many of these commands aside from the main upload or download action, this is not too practical.

OpenSSH doesn’t have a convenient interface for in-memory SFTP, libssh is under LGPL.

libssh2 provides a stable and user-friendly interface, and curl’s implementation doesn’t really add any significant business logic to that, so I would prefer to use libssh2 only, if not for the need for proxy support: libssh2 can't itself connect to a proxy.

curl's proxy support is excellent, on the other hand, and fortunately we can use CURLOPT_CONNECTONLY and CURLOPT_ACTIVESOCKET to sort out all the proxy operations and give us a socket to use afterwards. We can then give that socket to libssh2, and implement our SFTP functionality with it.

I think this is our best option, so I’ve created a standalone tool for testing it, that can use curl to connect to a proxy, then download and upload files through SFTP using libssh2, and it works with both SOCKS and HTTP (CONNECT) proxies.

Today I plan to start the work of integrating this into a processor.


was (Author: bakaid):
[~phrocker] 

I’ve looked at curl an other possible implementations (libssh, libssh2, OpenSSH). curl itself uses libssh2 internally. I’ve found that while we could probably provide most of the things we need using just curl, we wouldn’t have the control I would be comfortable with. curl works in a “one download or upload operation on a file or directory, and random commands before and after that” model: you specify one upload or download, and other commands to execute before that (CURLOPT_QUOTE) and after that (CURLOPT_POSTQUOTE). However, you don’t really have much control over these commands. You can either flag them as fatal, if failed, or non-fatal, if failed, but after executing the whole operation, you don’t know the result for each of these commands, and if one of them fails, you don’t know which one did. You also don’t have the range of parameterization available for commands which are available natively in the protocol and also provided by libssh2, and some of which I see would be useful in implementing the processors. Because we need to execute many of these commands aside from the main upload or download action, this is not too practical.

OpenSSH doesn’t have a convenient interface for in-memory SFTP, libssh is under LGPL.

libssh2 provides a stable and user-friendly interface, and curl’s implementation doesn’t really add any significant business logic to that, so I would prefer to use libssh2 only, if not for the need for proxy support: libssh2 can't itself connect to a proxy.

curl's proxy support is excellent, on the other hand, and fortunately we can use CURLOPT_CONNECTONLY and CURLOPT_ACTIVESOCKET to sort out all the proxy operations and give us a socket to use afterwards. We can then give that socket to libssh2, and implement our SFTP functionality with it.

I think this is our best option, so I’ve created a standalone tool for testing it, that can use curl to connect to a proxy, then download and upload files using through SFTP using libssh2, and it works with both SOCKS and HTTP (CONNECT) proxies.

Today I plan to start the work of integrating this into a processor.

> Add PutSFTP Support
> -------------------
>
>                 Key: MINIFICPP-542
>                 URL: https://issues.apache.org/jira/browse/MINIFICPP-542
>             Project: Apache NiFi MiNiFi C++
>          Issue Type: Bug
>            Reporter: Mr TheSegfault
>            Assignee: Daniel Bakai
>            Priority: Major
>              Labels: easyfix, newbie
>             Fix For: 0.7.0
>
>
> SFTP is an easy operation to add through the use of libCURL
>  
> Processors the can be added:
> (priority) PutSFTP
> ListSFTP
> FetchSFTP
> GetSFTP



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)