SMB/CIFS support in NiFi

[Si <si...@web.de>]

Hello

We're using NiFi on a Linux cluster because we get files from openssh/sftp
and then we send them to Windows shares. While it kind of works to just use
CIFS mounts, some of the Windows based systems are relying on share access
locks to know when it's safe to start reading the file. I could not get
this kind of locking to work using CIFS mounts. Supposedly the workaround
is to use a Windows cluster and S2S instead.

While looking for a solution I found an existing implementation for a
PutSMBFiles [1] processor. It depends on jcifs and that library does work
with the locks. But since jcifs is licensed under LGPL, I suppose there is
no chance to include it into the official NiFi.

There is an other lib though called smbj [2] and this is under Apache
License 2.0. Now my question is:
Is there already work ongoing for something like this?
If I'd start working on such a PutSMBFiles processor, how likely could this
be included into the official NiFi code base?
Is the put part enough for starting out or does it have to contain a get
(list / fetch) mechanism?
Any other considerations?

Regards
Si

[1]
https://github.com/dream-lab/echo/blob/master/misc/NifiTensorflowProcessors/src/main/java/in/dream_lab/nifi/tf/processors/PutSmbFiles.java
[2] https://github.com/hierynomus/smbj

Re: SMB/CIFS support in NiFi

[Pierre Villard <pi...@gmail.com>]

Hi Si,

Thanks for jumping in. I am not aware of any existing work on such a
processor, but it'd definitely be a nice addition. Feel free to open a pull
request with your processors when you feel it's ready for a review. IMHO,
you can open a pull request for a Put processor without the List/Fetch
couple.

Hope this helps,
Pierre

Le mar. 19 nov. 2019 à 14:08, Si <si...@web.de> a écrit :

> Well, here is the first version:
> https://github.com/si-sun/nifi-smb-processors
>
> It supports normal username / password authentication, missing dir
> creation, conflict resolution and share access setting.
>
> Regards
> Si
>
>
> Am Do., 14. Nov. 2019 um 19:20 Uhr schrieb Si <si...@web.de>:
>
> > Hello
> >
> > We're using NiFi on a Linux cluster because we get files from
> openssh/sftp
> > and then we send them to Windows shares. While it kind of works to just
> use
> > CIFS mounts, some of the Windows based systems are relying on share
> access
> > locks to know when it's safe to start reading the file. I could not get
> > this kind of locking to work using CIFS mounts. Supposedly the workaround
> > is to use a Windows cluster and S2S instead.
> >
> > While looking for a solution I found an existing implementation for a
> > PutSMBFiles [1] processor. It depends on jcifs and that library does work
> > with the locks. But since jcifs is licensed under LGPL, I suppose there
> is
> > no chance to include it into the official NiFi.
> >
> > There is an other lib though called smbj [2] and this is under Apache
> > License 2.0. Now my question is:
> > Is there already work ongoing for something like this?
> > If I'd start working on such a PutSMBFiles processor, how likely could
> > this be included into the official NiFi code base?
> > Is the put part enough for starting out or does it have to contain a get
> > (list / fetch) mechanism?
> > Any other considerations?
> >
> > Regards
> > Si
> >
> > [1]
> >
> https://github.com/dream-lab/echo/blob/master/misc/NifiTensorflowProcessors/src/main/java/in/dream_lab/nifi/tf/processors/PutSmbFiles.java
> > [2] https://github.com/hierynomus/smbj
> >
>

Re: SMB/CIFS support in NiFi

[Si <si...@web.de>]

Well, here is the first version:
https://github.com/si-sun/nifi-smb-processors

It supports normal username / password authentication, missing dir
creation, conflict resolution and share access setting.

Regards
Si


Am Do., 14. Nov. 2019 um 19:20 Uhr schrieb Si <si...@web.de>:

> Hello
>
> We're using NiFi on a Linux cluster because we get files from openssh/sftp
> and then we send them to Windows shares. While it kind of works to just use
> CIFS mounts, some of the Windows based systems are relying on share access
> locks to know when it's safe to start reading the file. I could not get
> this kind of locking to work using CIFS mounts. Supposedly the workaround
> is to use a Windows cluster and S2S instead.
>
> While looking for a solution I found an existing implementation for a
> PutSMBFiles [1] processor. It depends on jcifs and that library does work
> with the locks. But since jcifs is licensed under LGPL, I suppose there is
> no chance to include it into the official NiFi.
>
> There is an other lib though called smbj [2] and this is under Apache
> License 2.0. Now my question is:
> Is there already work ongoing for something like this?
> If I'd start working on such a PutSMBFiles processor, how likely could
> this be included into the official NiFi code base?
> Is the put part enough for starting out or does it have to contain a get
> (list / fetch) mechanism?
> Any other considerations?
>
> Regards
> Si
>
> [1]
> https://github.com/dream-lab/echo/blob/master/misc/NifiTensorflowProcessors/src/main/java/in/dream_lab/nifi/tf/processors/PutSmbFiles.java
> [2] https://github.com/hierynomus/smbj
>