[jira] [Created] (NIFI-7971) Create List-Fetch-Delete pattern

["Dennis Jaheruddin (Jira)" <ji...@apache.org>]

Dennis Jaheruddin created NIFI-7971:
---------------------------------------

             Summary: Create List-Fetch-Delete pattern
                 Key: NIFI-7971
                 URL: https://issues.apache.org/jira/browse/NIFI-7971
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
    Affects Versions: 1.12.1
            Reporter: Dennis Jaheruddin


*Challenge*

There are several processors for getting data. Originally we had GetHDFS, GetSFTP and these are now being phased out for List Fetch processors. 

I think the List-Fetch pattern is a significant improvement, but what I see now is that people are looking for a List-Fetch-Delete pattern. 

*Example usecase:*

Gather data, process and write in destination, delete

There are several ways to achieve this (native but inefficient ones, like 'do a second Fetch with remove source file and terminate success, or efficient but scripted ones, like 'use the executestreamprocessor to send the delete command').

As far as I understand, this would not be needed when using Stateless Nifi, but that is not relevant for many usecases.

*Proposal*

However, I propose we find a solution that allows people to easily achieve a List-Fetch-Delete pattern. 

Two lines of thought:
 # Expand Fetch processors with a 'deleteonly' mode, which would not actually copy in the data
 # Create dedicated processors, like 'DeleteSFTP' or 'ExecuteSFTP'

I have written the example for SFTP, but think this would be a natural extension for  most places where we have the List-Fetch pattern currently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)