[jira] [Comment Edited] (NIFI-25) Create processor set to support interaction with Amazon S3

["Mark Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341760#comment-14341760 ] 

Mark Payne edited comment on NIFI-25 at 3/14/15 9:11 PM:
---------------------------------------------------------

I'm not sure about the ListS3. I can definitely see the value of it. However, it requires that the processor maintain a significant amount of state about what it has seen This is not cluster friendly at all. It also requires continually pulling a potentially huge listing to see if anything has changed.

I think we should instead push users to configure S3 to add a notification to SQS when a new object is placed in an S3 bucket. We can then have a GetSQS processor to detect that an item was added and then fetch the contents via GetS3/FetchS3/RetrieveS3. This is a much more scalable approach and handles backpressure well.


was (Author: markap14):
I'm not sure about the ListS3. I can definitely see the value of it. However, it requires that the processor maintain a significant amount of state about what it has seen This is not cluster friendly at all. It also requires continually pulling a potentially huge listing to see if anything has changed.

I think we should instead push users to configure S3 to add a notification to SNS when a new object is placed in an S3 bucket. We can then have a GetSNS processor to detect that an item was added and then fetch the contents via GetS3/FetchS3/RetrieveS3. This is a much more scalable approach and handles backpressure well.

> Create processor set to support interaction with Amazon S3
> ----------------------------------------------------------
>
>                 Key: NIFI-25
>                 URL: https://issues.apache.org/jira/browse/NIFI-25
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: Joseph Witt
>            Priority: Minor
>
> Determine the appropriate set of processors to interact fully with Amazon's S3.  
> Might need:
> - ListS3
> - RetrieveS3 or GetS3
> - PutS3
> May be able to replace all of these simply with combinations of use of InvokeHTTP.  But need to ensure a high quality user experience as well so it isn't only a power user feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)