[jira] [Updated] (NIFI-7971) List-Fetch-Delete pattern

["Dennis Jaheruddin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dennis Jaheruddin updated NIFI-7971:
------------------------------------
    Summary: List-Fetch-Delete pattern  (was: Create List-Fetch-Delete pattern)

> List-Fetch-Delete pattern
> -------------------------
>
>                 Key: NIFI-7971
>                 URL: https://issues.apache.org/jira/browse/NIFI-7971
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.12.1
>            Reporter: Dennis Jaheruddin
>            Priority: Minor
>
> *Challenge*
> There are several processors for getting data. Originally we had GetHDFS, GetSFTP and these are now being phased out for List Fetch processors. 
> I think the List-Fetch pattern is a significant improvement, but what I see now is that people are looking for a List-Fetch-Delete pattern. 
> *Example usecase:*
> Gather data, process and write in destination, delete
> There are several ways to achieve this (native but inefficient ones, like 'do a second Fetch with remove source file and terminate success', or efficient but scripted ones, like 'use the executestreamprocessor to send the delete command').
> As far as I understand, this would not be needed when using Stateless Nifi, but that is not relevant for many usecases.
> *Proposal*
> However, I propose we find a solution that allows people to easily achieve a List-Fetch-Delete pattern. 
> Two lines of thought:
>  # Expand Fetch processors with a 'deleteonly' mode, which would not actually copy in the data
>  # Create dedicated processors, like 'DeleteSFTP' or 'ExecuteSFTP'
> I have written the example for SFTP, but think this would be a natural extension for  most places where we have the List-Fetch pattern currently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)