You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Ravi Papisetti (rpapiset)" <rp...@cisco.com> on 2017/12/06 07:14:07 UTC

NiFi copying windows .part files

Hi,

We are using Apache NiFi 1.3.0

We have a process flow to copy files from NFS to HDFS (with processors ListFile, FetchFile and PutHDFS)

In the NiFi process flow, ListFile is configured to listen to a directory on NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS (while transfer is in the middle), a part file(x.csv.part) is created at NFS until transfer is complete.

ListFile has picked up this x.csv.part file and fetchFile picked up this to transfer to HDFS, didn’t update the file name back to x.csv in HDFS when transfer is complete.

But, in case a file from linux file system, while file copy to NFS is in progress it created (.x.csv) and when transfer is complete, at both NFS and HDFS, filename is updated to x.csv (from .x.csv).

Any thought how we can configure ListFile not to pickup these part files or any configurations in NiFi that fixes file names for these windows part files?

Appreciate your help.

Thanks,
Ravi Papisetti

Re: NiFi copying windows .part files

Posted by "Ravi Papisetti (rpapiset)" <rp...@cisco.com>.
File names are ending with .part (.part is getting suffixed to filename, not prefixed).

In case of files from linux file system, “.” (dot) getting prefixed while transfer is in progress.

Thanks,
Ravi Papisetti

On 06/12/17, 1:47 AM, "Joe Witt" <jo...@gmail.com> wrote:

    Imagine a filename construct where you wanted to pick up any file that
    begins with the phrase 'start' but does NOT end in the phrase 'part'.
    
    The name is of a form 'begin.middle.end'.
    
    This filename start.middle.ok would get picked up.
    
    This filename start.middle.part would not.
    
    The pattern for that example would be
      start\..+\.(?!part)
    
    The key part of that is the negative lookahead for ensuring it does
    not end in part.
    
    Thanks
    
    On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset)
    <rp...@cisco.com> wrote:
    > Yeah..that is good idea, but we are already using this option to copy file with certain prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion.
    >
    > Any thoughts.
    >
    > Thanks,
    > Ravi Papisetti
    >
    > On 06/12/17, 1:26 AM, "Joe Witt" <jo...@gmail.com> wrote:
    >
    >     Ravi
    >
    >     Please use the 'File Filter' property of ListFile to control ignoring
    >     filenames until they no longer end in 'part'.
    >
    >     Thanks
    >
    >     On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
    >     <rp...@cisco.com> wrote:
    >     > Hi,
    >     >
    >     >
    >     >
    >     > We are using Apache NiFi 1.3.0
    >     >
    >     >
    >     >
    >     > We have a process flow to copy files from NFS to HDFS (with processors
    >     > ListFile, FetchFile and PutHDFS)
    >     >
    >     >
    >     >
    >     > In the NiFi process flow, ListFile is configured to listen to a directory on
    >     > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
    >     > (while transfer is in the middle), a part file(x.csv.part) is created at NFS
    >     > until transfer is complete.
    >     >
    >     >
    >     >
    >     > ListFile has picked up this x.csv.part file and fetchFile picked up this to
    >     > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
    >     > transfer is complete.
    >     >
    >     >
    >     >
    >     > But, in case a file from linux file system, while file copy to NFS is in
    >     > progress it created (.x.csv) and when transfer is complete, at both NFS and
    >     > HDFS, filename is updated to x.csv (from .x.csv).
    >     >
    >     >
    >     >
    >     > Any thought how we can configure ListFile not to pickup these part files or
    >     > any configurations in NiFi that fixes file names for these windows part
    >     > files?
    >     >
    >     >
    >     >
    >     > Appreciate your help.
    >     >
    >     >
    >     >
    >     > Thanks,
    >     >
    >     > Ravi Papisetti
    >
    >

Re: NiFi copying windows .part files

Posted by Joe Witt <jo...@gmail.com>.
Imagine a filename construct where you wanted to pick up any file that
begins with the phrase 'start' but does NOT end in the phrase 'part'.

The name is of a form 'begin.middle.end'.

This filename start.middle.ok would get picked up.

This filename start.middle.part would not.

The pattern for that example would be
  start\..+\.(?!part)

The key part of that is the negative lookahead for ensuring it does
not end in part.

Thanks

On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset)
<rp...@cisco.com> wrote:
> Yeah..that is good idea, but we are already using this option to copy file with certain prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion.
>
> Any thoughts.
>
> Thanks,
> Ravi Papisetti
>
> On 06/12/17, 1:26 AM, "Joe Witt" <jo...@gmail.com> wrote:
>
>     Ravi
>
>     Please use the 'File Filter' property of ListFile to control ignoring
>     filenames until they no longer end in 'part'.
>
>     Thanks
>
>     On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
>     <rp...@cisco.com> wrote:
>     > Hi,
>     >
>     >
>     >
>     > We are using Apache NiFi 1.3.0
>     >
>     >
>     >
>     > We have a process flow to copy files from NFS to HDFS (with processors
>     > ListFile, FetchFile and PutHDFS)
>     >
>     >
>     >
>     > In the NiFi process flow, ListFile is configured to listen to a directory on
>     > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
>     > (while transfer is in the middle), a part file(x.csv.part) is created at NFS
>     > until transfer is complete.
>     >
>     >
>     >
>     > ListFile has picked up this x.csv.part file and fetchFile picked up this to
>     > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
>     > transfer is complete.
>     >
>     >
>     >
>     > But, in case a file from linux file system, while file copy to NFS is in
>     > progress it created (.x.csv) and when transfer is complete, at both NFS and
>     > HDFS, filename is updated to x.csv (from .x.csv).
>     >
>     >
>     >
>     > Any thought how we can configure ListFile not to pickup these part files or
>     > any configurations in NiFi that fixes file names for these windows part
>     > files?
>     >
>     >
>     >
>     > Appreciate your help.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Ravi Papisetti
>
>

Re: NiFi copying windows .part files

Posted by "Ravi Papisetti (rpapiset)" <rp...@cisco.com>.
Yeah..that is good idea, but we are already using this option to copy file with certain prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion.

Any thoughts.

Thanks,
Ravi Papisetti

On 06/12/17, 1:26 AM, "Joe Witt" <jo...@gmail.com> wrote:

    Ravi
    
    Please use the 'File Filter' property of ListFile to control ignoring
    filenames until they no longer end in 'part'.
    
    Thanks
    
    On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
    <rp...@cisco.com> wrote:
    > Hi,
    >
    >
    >
    > We are using Apache NiFi 1.3.0
    >
    >
    >
    > We have a process flow to copy files from NFS to HDFS (with processors
    > ListFile, FetchFile and PutHDFS)
    >
    >
    >
    > In the NiFi process flow, ListFile is configured to listen to a directory on
    > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
    > (while transfer is in the middle), a part file(x.csv.part) is created at NFS
    > until transfer is complete.
    >
    >
    >
    > ListFile has picked up this x.csv.part file and fetchFile picked up this to
    > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
    > transfer is complete.
    >
    >
    >
    > But, in case a file from linux file system, while file copy to NFS is in
    > progress it created (.x.csv) and when transfer is complete, at both NFS and
    > HDFS, filename is updated to x.csv (from .x.csv).
    >
    >
    >
    > Any thought how we can configure ListFile not to pickup these part files or
    > any configurations in NiFi that fixes file names for these windows part
    > files?
    >
    >
    >
    > Appreciate your help.
    >
    >
    >
    > Thanks,
    >
    > Ravi Papisetti

Re: NiFi copying windows .part files

Posted by Joe Witt <jo...@gmail.com>.
Ravi

Please use the 'File Filter' property of ListFile to control ignoring
filenames until they no longer end in 'part'.

Thanks

On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
<rp...@cisco.com> wrote:
> Hi,
>
>
>
> We are using Apache NiFi 1.3.0
>
>
>
> We have a process flow to copy files from NFS to HDFS (with processors
> ListFile, FetchFile and PutHDFS)
>
>
>
> In the NiFi process flow, ListFile is configured to listen to a directory on
> NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
> (while transfer is in the middle), a part file(x.csv.part) is created at NFS
> until transfer is complete.
>
>
>
> ListFile has picked up this x.csv.part file and fetchFile picked up this to
> transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
> transfer is complete.
>
>
>
> But, in case a file from linux file system, while file copy to NFS is in
> progress it created (.x.csv) and when transfer is complete, at both NFS and
> HDFS, filename is updated to x.csv (from .x.csv).
>
>
>
> Any thought how we can configure ListFile not to pickup these part files or
> any configurations in NiFi that fixes file names for these windows part
> files?
>
>
>
> Appreciate your help.
>
>
>
> Thanks,
>
> Ravi Papisetti