[jira] [Created] (NIFI-3449) Create Google Cloud Platform/Google Cloud Storage Processors

["Gene Peters (JIRA)" <ji...@apache.org>]

Gene Peters created NIFI-3449:
---------------------------------

             Summary: Create Google Cloud Platform/Google Cloud Storage Processors
                 Key: NIFI-3449
                 URL: https://issues.apache.org/jira/browse/NIFI-3449
             Project: Apache NiFi
          Issue Type: New Feature
          Components: Extensions
            Reporter: Gene Peters


Hi all,

We had a need in our production deployments to interact with Google Cloud Storage. At the time, NIFI-2809 hadn't seen much movement, and after applying the patch I found that the configuration was too specific for my needs (it's hardcoded to use "Application Default" credentials, everything uploaded to GCS is uploaded with the "public" ACL, etc). So I created a series of Processors / Controller Services based off of the AWS NiFi library, and would like to contribute them. 

Features:
* All credentialing is handled by a controller service, allowing multiple processors to use the same service / credentials
* An Abstract processor is provided which forms the basis for all GCP related processors.
* The standard Google Cloud Storage operations are supported, very similarly to the AWS S3 processors: ListGCSBucket, DeleteGCSObject, FetchGCSObject, PutGCSObject
* Everything is documented and unit tested. 
* I've also provided integration tests, but they're disabled by default (as they require Google Cloud credentials). To run them, use the flag {{skipGCPIntegrationTests=false}}

Todo:
* The GCP Java library's ReadChannel objects implement the "restorable" interface, which allows for state saving / checkpointing. I'd really like to leverage this with the State support that NiFi provides, but it would require serializing / deserializing the object. 

I'm going to be submitting this as a pull request through GitHub. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)