You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mikhail Sosonkin <mi...@synack.com> on 2017/02/27 02:27:35 UTC
Processors for Google Cloud Platform
Hello Everyone,
SYNACK has recently started working with Google Cloud and we use NiFi to
move some of our data sets. For that, we've built processors to work with
Cloud Storage and Pubsub systems.
A processor that uploads messages to Google Cloud Platform PubSub topic:
https://github.com/synack/nifi-gcp-pubsub-publisher
A processor that reads messages from a Google Cloud Platform PubSub topic:
https://github.com/synack/nifi-gcp-pubsub-consumer
A processor that uploads files to Google Cloud Platform Cloud: Storage:
https://github.com/synack/nifi-gcp-putgcs
They are certainly works in progress, but we're running them in our
production environments inside and outside of GCP without any issues. Since
these processors have been working so well for us, I wanted to share them
with the community - just in case some one else finds them useful.
Our team is still growing and being able to do these functions as part of
NiFi, rather than building specialized widgets, has been a major boon for
us. So, thanks NIFI team!
Mike.
http://synack.com
--
This email may contain material that is confidential for the sole use of
the intended recipient(s). Any review, reliance or distribution or
disclosure by others without express permission is strictly prohibited. If
you are not the intended recipient, please contact the sender and delete
all copies of this message.
Re: Processors for Google Cloud Platform
Posted by Andre <an...@fucs.org>.
Mikhail,
The definitive source would be
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Providingcodeordocumentationcontributions
But it is more or less:
1. Search for an existing JIRA ticket covering the work
2. If none exists, touch base with the overall dev / user community (you
did that already)
3. Open the ticket and/or ask it to be assigned to you
4. Creating a git branch of the main NiFi tree (either via git or Github
https://github.com/apache/nifi )
5. Code! Code! Code!
6. Submit a patch (JIRA) or PR (Github)
7. Action on feedback provided during the peer view of your code until it
gets ready to be merged
8. Be proud when you commit get merged (and share on twitter that your
contribution has been immortalised by the git log!!!!. :-) )
Cheers
PS-I also noticed that although you cross posted, my reply only went to
users, so shifting it back to dev as that is a more appropriate forum.
On Mon, Feb 27, 2017 at 3:30 PM, Mikhail Sosonkin <mi...@synack.com>
wrote:
> Andre,
>
> I would love to get the PubSub processors into nifi proper. I'm not too
> familiar with your development processes, can you give me some directions
> on how you want me to do that and what sort of things the code should have?
>
> Thanks,
> Mike.
>
> On Sun, Feb 26, 2017 at 9:58 PM, Andre <an...@fucs.org> wrote:
>
>> Mikhail,
>>
>> Thanks for the email!
>>
>> Someone had recently contributed processors to handle Google Cloud
>> Storage buckets and objects, however I am not aware of anyone providing
>> processors to handle PubSub yet so I am sure it will be a welcome processor!
>>
>> Would you be willing to submit a PR to include your processors as part of
>> the GCS bundle?
>>
>> Cheers
>>
>> On Mon, Feb 27, 2017 at 1:27 PM, Mikhail Sosonkin <mi...@synack.com>
>> wrote:
>>
>>> Hello Everyone,
>>>
>>> SYNACK has recently started working with Google Cloud and we use NiFi to
>>> move some of our data sets. For that, we've built processors to work with
>>> Cloud Storage and Pubsub systems.
>>>
>>> A processor that uploads messages to Google Cloud Platform PubSub topic:
>>> https://github.com/synack/nifi-gcp-pubsub-publisher
>>>
>>> A processor that reads messages from a Google Cloud Platform PubSub
>>> topic: https://github.com/synack/nifi-gcp-pubsub-consumer
>>>
>>> A processor that uploads files to Google Cloud Platform Cloud: Storage:
>>> https://github.com/synack/nifi-gcp-putgcs
>>>
>>> They are certainly works in progress, but we're running them in our
>>> production environments inside and outside of GCP without any issues. Since
>>> these processors have been working so well for us, I wanted to share them
>>> with the community - just in case some one else finds them useful.
>>>
>>> Our team is still growing and being able to do these functions as part
>>> of NiFi, rather than building specialized widgets, has been a major boon
>>> for us. So, thanks NIFI team!
>>>
>>> Mike.
>>> http://synack.com
>>>
>>> This email may contain material that is confidential for the sole use of
>>> the intended recipient(s). Any review, reliance or distribution or
>>> disclosure by others without express permission is strictly prohibited. If
>>> you are not the intended recipient, please contact the sender and delete
>>> all copies of this message.
>>>
>>
>>
>
> This email may contain material that is confidential for the sole use of
> the intended recipient(s). Any review, reliance or distribution or
> disclosure by others without express permission is strictly prohibited. If
> you are not the intended recipient, please contact the sender and delete
> all copies of this message.
>
Re: Processors for Google Cloud Platform
Posted by Mikhail Sosonkin <mi...@synack.com>.
Andre,
I would love to get the PubSub processors into nifi proper. I'm not too
familiar with your development processes, can you give me some directions
on how you want me to do that and what sort of things the code should have?
Thanks,
Mike.
On Sun, Feb 26, 2017 at 9:58 PM, Andre <an...@fucs.org> wrote:
> Mikhail,
>
> Thanks for the email!
>
> Someone had recently contributed processors to handle Google Cloud Storage
> buckets and objects, however I am not aware of anyone providing processors
> to handle PubSub yet so I am sure it will be a welcome processor!
>
> Would you be willing to submit a PR to include your processors as part of
> the GCS bundle?
>
> Cheers
>
> On Mon, Feb 27, 2017 at 1:27 PM, Mikhail Sosonkin <mi...@synack.com>
> wrote:
>
>> Hello Everyone,
>>
>> SYNACK has recently started working with Google Cloud and we use NiFi to
>> move some of our data sets. For that, we've built processors to work with
>> Cloud Storage and Pubsub systems.
>>
>> A processor that uploads messages to Google Cloud Platform PubSub topic:
>> https://github.com/synack/nifi-gcp-pubsub-publisher
>>
>> A processor that reads messages from a Google Cloud Platform PubSub
>> topic: https://github.com/synack/nifi-gcp-pubsub-consumer
>>
>> A processor that uploads files to Google Cloud Platform Cloud: Storage:
>> https://github.com/synack/nifi-gcp-putgcs
>>
>> They are certainly works in progress, but we're running them in our
>> production environments inside and outside of GCP without any issues. Since
>> these processors have been working so well for us, I wanted to share them
>> with the community - just in case some one else finds them useful.
>>
>> Our team is still growing and being able to do these functions as part of
>> NiFi, rather than building specialized widgets, has been a major boon for
>> us. So, thanks NIFI team!
>>
>> Mike.
>> http://synack.com
>>
>> This email may contain material that is confidential for the sole use of
>> the intended recipient(s). Any review, reliance or distribution or
>> disclosure by others without express permission is strictly prohibited. If
>> you are not the intended recipient, please contact the sender and delete
>> all copies of this message.
>>
>
>
--
This email may contain material that is confidential for the sole use of
the intended recipient(s). Any review, reliance or distribution or
disclosure by others without express permission is strictly prohibited. If
you are not the intended recipient, please contact the sender and delete
all copies of this message.
Re: Processors for Google Cloud Platform
Posted by Andre <an...@fucs.org>.
Mikhail,
Thanks for the email!
Someone had recently contributed processors to handle Google Cloud Storage
buckets and objects, however I am not aware of anyone providing processors
to handle PubSub yet so I am sure it will be a welcome processor!
Would you be willing to submit a PR to include your processors as part of
the GCS bundle?
Cheers
On Mon, Feb 27, 2017 at 1:27 PM, Mikhail Sosonkin <mi...@synack.com>
wrote:
> Hello Everyone,
>
> SYNACK has recently started working with Google Cloud and we use NiFi to
> move some of our data sets. For that, we've built processors to work with
> Cloud Storage and Pubsub systems.
>
> A processor that uploads messages to Google Cloud Platform PubSub topic:
> https://github.com/synack/nifi-gcp-pubsub-publisher
>
> A processor that reads messages from a Google Cloud Platform PubSub topic:
> https://github.com/synack/nifi-gcp-pubsub-consumer
>
> A processor that uploads files to Google Cloud Platform Cloud: Storage:
> https://github.com/synack/nifi-gcp-putgcs
>
> They are certainly works in progress, but we're running them in our
> production environments inside and outside of GCP without any issues. Since
> these processors have been working so well for us, I wanted to share them
> with the community - just in case some one else finds them useful.
>
> Our team is still growing and being able to do these functions as part of
> NiFi, rather than building specialized widgets, has been a major boon for
> us. So, thanks NIFI team!
>
> Mike.
> http://synack.com
>
> This email may contain material that is confidential for the sole use of
> the intended recipient(s). Any review, reliance or distribution or
> disclosure by others without express permission is strictly prohibited. If
> you are not the intended recipient, please contact the sender and delete
> all copies of this message.
>