How to lock getfile upto putfile write into same file?

[prabhu Mahendran <pr...@gmail.com>]

I have scheduled getfile processor to 0 sec to track the local folder.

Issue I have faced: PutFile is appending few flowfiles into single file.
Getfile has been configured with keepsourcefile as false. So getfile is
fetching partial content before putfile writes into local location.

How to handle this issue? Can we lock the file till putfile/custom
processor completely writes the file and remove lock once completed??

Re: How to lock getfile upto putfile write into same file?

[prabhu Mahendran <pr...@gmail.com>]

I have running multiple instances of PutFile writing to same file that
leads fail in Strange ways.

I have tried MergeContent processor but it not yields same results for all
time.

For one time it correctly merges files according to the filename but
sometimes it could not merge.

In my use case i have to store the contents in 10 files(number may be
changed).A filename is specified in every flow files.

I couldn't  merge the files with same filename in some times.And also i
cannot use expression language in "Minimum Number of entries" attribute if
i have set that property directly it can work.



On Thu, May 18, 2017 at 2:09 PM, Joe Witt <jo...@gmail.com> wrote:

> PutFile while writing automatically writes with a dot prepended and
> once the write is complete it removes the dot.
>
> If you have multiple instances of PutFile writing to the same file
> with the intent of appending to that file it will like fail in strange
> ways.
>
> I would recommend you simply build up the complete thing you want to
> write out to a file using MergeContent and send the merged results to
> PutFile.  However, it should not be the case that you have PutFile and
> GetFile sharing a given file in the same NiFi as you can simply
> connect the processors to move the data without have to leave the
> confines of NiFi using file IO.
>
> Thanks
>
> On Thu, May 18, 2017 at 3:06 AM, prabhu Mahendran
> <pr...@gmail.com> wrote:
> > Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
> > name(without '.') will automatically changed by Putfile when its done?
> >
> >
> >
> > Since I am appending each rows from html content into proper csv using
> > Putfile processor. Is this works fine? How putfile knows complete html
> > flowfiles has been moved.
> >
> >
> > On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros <he...@gmail.com>
> wrote:
> >>
> >> Prabhu,
> >>
> >> PutFile should pre-pend files with a "." while it is writing to it and
> >> then in the end will copy the "." file to its "filename" value.
> >>
> >> On GetFile side make sure that "ignore hidden files" is set to true.
> >>
> >>
> >>
> >> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran <
> prabhuu161994@gmail.com>
> >> wrote:
> >>>
> >>> I have scheduled getfile processor to 0 sec to track the local folder.
> >>>
> >>> Issue I have faced: PutFile is appending few flowfiles into single
> file.
> >>> Getfile has been configured with keepsourcefile as false. So getfile is
> >>> fetching partial content before putfile writes into local location.
> >>>
> >>> How to handle this issue? Can we lock the file till putfile/custom
> >>> processor completely writes the file and remove lock once completed??
> >
> >
>

Re: How to lock getfile upto putfile write into same file?

[Joe Witt <jo...@gmail.com>]

PutFile while writing automatically writes with a dot prepended and
once the write is complete it removes the dot.

If you have multiple instances of PutFile writing to the same file
with the intent of appending to that file it will like fail in strange
ways.

I would recommend you simply build up the complete thing you want to
write out to a file using MergeContent and send the merged results to
PutFile.  However, it should not be the case that you have PutFile and
GetFile sharing a given file in the same NiFi as you can simply
connect the processors to move the data without have to leave the
confines of NiFi using file IO.

Thanks

On Thu, May 18, 2017 at 3:06 AM, prabhu Mahendran
<pr...@gmail.com> wrote:
> Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
> name(without '.') will automatically changed by Putfile when its done?
>
>
>
> Since I am appending each rows from html content into proper csv using
> Putfile processor. Is this works fine? How putfile knows complete html
> flowfiles has been moved.
>
>
> On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros <he...@gmail.com> wrote:
>>
>> Prabhu,
>>
>> PutFile should pre-pend files with a "." while it is writing to it and
>> then in the end will copy the "." file to its "filename" value.
>>
>> On GetFile side make sure that "ignore hidden files" is set to true.
>>
>>
>>
>> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran <pr...@gmail.com>
>> wrote:
>>>
>>> I have scheduled getfile processor to 0 sec to track the local folder.
>>>
>>> Issue I have faced: PutFile is appending few flowfiles into single file.
>>> Getfile has been configured with keepsourcefile as false. So getfile is
>>> fetching partial content before putfile writes into local location.
>>>
>>> How to handle this issue? Can we lock the file till putfile/custom
>>> processor completely writes the file and remove lock once completed??
>
>

Re: How to lock getfile upto putfile write into same file?

[prabhu Mahendran <pr...@gmail.com>]

Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
name(without '.') will automatically changed by Putfile when its done?



Since I am appending each rows from html content into proper csv using
Putfile processor. Is this works fine? How putfile knows complete html
flowfiles has been moved.

On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros <he...@gmail.com> wrote:

> Prabhu,
>
> PutFile should pre-pend files with a "." while it is writing to it and
> then in the end will copy the "." file to its "filename" value.
>
> On GetFile side make sure that "ignore hidden files" is set to true.
>
>
>
> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran <pr...@gmail.com>
> wrote:
>
>> I have scheduled getfile processor to 0 sec to track the local folder.
>>
>> Issue I have faced: PutFile is appending few flowfiles into single file.
>> Getfile has been configured with keepsourcefile as false. So getfile is
>> fetching partial content before putfile writes into local location.
>>
>> How to handle this issue? Can we lock the file till putfile/custom
>> processor completely writes the file and remove lock once completed??
>>
>

Re: How to lock getfile upto putfile write into same file?

[Juan Sequeiros <he...@gmail.com>]

Prabhu,

PutFile should pre-pend files with a "." while it is writing to it and then
in the end will copy the "." file to its "filename" value.

On GetFile side make sure that "ignore hidden files" is set to true.



On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran <pr...@gmail.com>
wrote:

> I have scheduled getfile processor to 0 sec to track the local folder.
>
> Issue I have faced: PutFile is appending few flowfiles into single file.
> Getfile has been configured with keepsourcefile as false. So getfile is
> fetching partial content before putfile writes into local location.
>
> How to handle this issue? Can we lock the file till putfile/custom
> processor completely writes the file and remove lock once completed??
>