You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Shawn Weeks <sw...@weeksconsulting.us> on 2018/10/15 20:10:20 UTC
FetchFile Completion Strategy
I was just wondering if it's expected behavior for the FetchFile Processor to eat errors related to the completion strategy. If for example your completion strategy is move and you specify a directory that doesn't exist or that NiFi doesn't have permissions on I'd have expected the Processor to route to failure not success.
Thanks
Shawn Weeks
Re: FetchFile Completion Strategy
Posted by Bryan Bende <bb...@gmail.com>.
It is possible the directory could get created and then something else
fails, but personally I can't think of any downside to that, and seems
like a better trade-off than routing to success, but if anyone else
has thoughts please let me know.
On Wed, Oct 17, 2018 at 12:07 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
>
> Took a look at the pull request and that should handle the issue I was seeing. Is there going to be an issue with the directory being left there if something else fails? For my case the empty directory is fine.
>
>
> Thanks
>
> Shawn Weeks
>
> ________________________________
> From: Bryan Bende <bb...@gmail.com>
> Sent: Wednesday, October 17, 2018 10:48:20 AM
> To: users@nifi.apache.org
> Subject: Re: FetchFile Completion Strategy
>
> Hi Shawn,
>
> The way the processor works is the following...
>
> 1) Perform checks to try and ensure the completion strategy will work later
> 2) Perform the fetch and if successful then transfer to success and
> commit the session
> 3) Perform the completion strategy
>
> The issue is if the completion fails in step 3, it is too late to
> route to failure because in step 2 the session was already committed.
>
> The reason it is done this way is because if we fetched the file
> without committing the session and then executed the completion
> strategy, it would be possible NiFi could crash at this moment before
> committing the session, and now the source file could be gone from the
> completion strategy and the data is also gone from NiFi since it was
> never committed, so there would be data loss.
>
> All that being said, I think I was able to figure out an improvement
> to protect against the case where there is a permission problem...
>
> https://github.com/apache/nifi/pull/3088
>
> Thanks,
>
> Bryan
>
> On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
> >
> > I was just wondering if it's expected behavior for the FetchFile Processor to eat errors related to the completion strategy. If for example your completion strategy is move and you specify a directory that doesn't exist or that NiFi doesn't have permissions on I'd have expected the Processor to route to failure not success.
> >
> >
> > Thanks
> >
> > Shawn Weeks
Re: FetchFile Completion Strategy
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Took a look at the pull request and that should handle the issue I was seeing. Is there going to be an issue with the directory being left there if something else fails? For my case the empty directory is fine.
Thanks
Shawn Weeks
________________________________
From: Bryan Bende <bb...@gmail.com>
Sent: Wednesday, October 17, 2018 10:48:20 AM
To: users@nifi.apache.org
Subject: Re: FetchFile Completion Strategy
Hi Shawn,
The way the processor works is the following...
1) Perform checks to try and ensure the completion strategy will work later
2) Perform the fetch and if successful then transfer to success and
commit the session
3) Perform the completion strategy
The issue is if the completion fails in step 3, it is too late to
route to failure because in step 2 the session was already committed.
The reason it is done this way is because if we fetched the file
without committing the session and then executed the completion
strategy, it would be possible NiFi could crash at this moment before
committing the session, and now the source file could be gone from the
completion strategy and the data is also gone from NiFi since it was
never committed, so there would be data loss.
All that being said, I think I was able to figure out an improvement
to protect against the case where there is a permission problem...
https://github.com/apache/nifi/pull/3088
Thanks,
Bryan
On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
>
> I was just wondering if it's expected behavior for the FetchFile Processor to eat errors related to the completion strategy. If for example your completion strategy is move and you specify a directory that doesn't exist or that NiFi doesn't have permissions on I'd have expected the Processor to route to failure not success.
>
>
> Thanks
>
> Shawn Weeks
Re: FetchFile Completion Strategy
Posted by Bryan Bende <bb...@gmail.com>.
Hi Shawn,
The way the processor works is the following...
1) Perform checks to try and ensure the completion strategy will work later
2) Perform the fetch and if successful then transfer to success and
commit the session
3) Perform the completion strategy
The issue is if the completion fails in step 3, it is too late to
route to failure because in step 2 the session was already committed.
The reason it is done this way is because if we fetched the file
without committing the session and then executed the completion
strategy, it would be possible NiFi could crash at this moment before
committing the session, and now the source file could be gone from the
completion strategy and the data is also gone from NiFi since it was
never committed, so there would be data loss.
All that being said, I think I was able to figure out an improvement
to protect against the case where there is a permission problem...
https://github.com/apache/nifi/pull/3088
Thanks,
Bryan
On Mon, Oct 15, 2018 at 4:10 PM Shawn Weeks <sw...@weeksconsulting.us> wrote:
>
> I was just wondering if it's expected behavior for the FetchFile Processor to eat errors related to the completion strategy. If for example your completion strategy is move and you specify a directory that doesn't exist or that NiFi doesn't have permissions on I'd have expected the Processor to route to failure not success.
>
>
> Thanks
>
> Shawn Weeks