You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jeff Zemerick <jz...@apache.org> on 2017/04/20 14:06:23 UTC
MiNiFi's differentiator
Hi all,
MiNiFi's WholeConfigDifferentiator along with the PullHttpChangeIngestor is
working well for me. I see that the differentiator is configurable and
additional implementations can be provided. Are there any examples of
circumstances in which the using WholeConfigDifferentiator would not be the
best choice?
Thanks,
Jeff
RE: [EXTERNAL] - Re: GetTwitter to stream tweets instead pull
Posted by Jamie Wang <ja...@opentext.com>.
Hi Joey,
Thanks for the information. The name actually plays only a small part as you indicated. But I sort of got convinced it is pulling after reading the help documentation for GetTwitter. You can see it here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.twitter.GetTwitter/. The line "Pulls status changes from Twitter's streaming API" sort of got me to believe it is a pull instead streaming. Also it'd be a good idea if possible to add a line to explicitly document it is actually streaming. Thanks again for your note and my apology for the late response.
Jamie
From: Joey Frazee [mailto:joey.frazee@icloud.com]
Sent: Wednesday, May 03, 2017 12:51 PM
To: users@nifi.apache.org
Subject: [EXTERNAL] - Re: GetTwitter to stream tweets instead pull
Jamie, can you explain a little bit more about what you’re looking for?
The GetTwitter processor is accessing the spritzer/decahose/firehouse, what have you, via Twitter’s Hosebird library. This library is indeed streaming the Tweets from their sample and filter APIs in the usual way with a persistent, chunk-encoded HTTP connection to https://stream.twitter.com/1.1/statuses/sample.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__stream.twitter.com_1.1_statuses_sample.json&d=DwMFaQ&c=ZgVRmm3mf2P1-XDAyDsu4A&r=TauVD_Op4rvIkArzdRrvTf-yuf4tmnM8R0LshdIDMbA&m=6M-GKh1Kc9QCdlSt4lVirUcS1suuGRF4MMEfHvjR45A&s=a8suuN1dyAV6Q9X5Z0tEns7WODrBD8VcmH0ctYLhJto&e=> and https://stream.twitter.com/1.1/statuses/filter.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__stream.twitter.com_1.1_statuses_filter.json&d=DwMFaQ&c=ZgVRmm3mf2P1-XDAyDsu4A&r=TauVD_Op4rvIkArzdRrvTf-yuf4tmnM8R0LshdIDMbA&m=6M-GKh1Kc9QCdlSt4lVirUcS1suuGRF4MMEfHvjR45A&s=0TueQgZE0p1YdPjGiDCQ5IVHtDBL5h_C0DZsOFKpEsA&e=>.
I’ll admit the name might be a little confusing since the Get might suggest it’s hitting one of the REST https://api.twitter.com/1.1/statuses/<https://urldefense.proofpoint.com/v2/url?u=https-3A__api.twitter.com_1.1_statuses_&d=DwMFaQ&c=ZgVRmm3mf2P1-XDAyDsu4A&r=TauVD_Op4rvIkArzdRrvTf-yuf4tmnM8R0LshdIDMbA&m=6M-GKh1Kc9QCdlSt4lVirUcS1suuGRF4MMEfHvjR45A&s=-8N0_UEmgW8Yb56o43fS87OzYVjtoFlZmSBZzVTfa1Y&e=> resources periodically instead of using a long-term HTTP connection.
-joey
On May 3, 2017, at 2:00 PM, Jamie Wang <ja...@opentext.com>> wrote:
Hi,
I understand the built-in processor GetTwitter is a pull. Are there streaming based processor for getting Tweets available? If no, any suggestions on how would l go by to build one?
Thanks
Jamie
Re: GetTwitter to stream tweets instead pull
Posted by Joey Frazee <jo...@icloud.com>.
Jamie, can you explain a little bit more about what you’re looking for?
The GetTwitter processor is accessing the spritzer/decahose/firehouse, what have you, via Twitter’s Hosebird library. This library is indeed streaming the Tweets from their sample and filter APIs in the usual way with a persistent, chunk-encoded HTTP connection to https://stream.twitter.com/1.1/statuses/sample.json and https://stream.twitter.com/1.1/statuses/filter.json.
I’ll admit the name might be a little confusing since the Get might suggest it’s hitting one of the REST https://api.twitter.com/1.1/statuses/ resources periodically instead of using a long-term HTTP connection.
-joey
> On May 3, 2017, at 2:00 PM, Jamie Wang <ja...@opentext.com> wrote:
>
> Hi,
>
> I understand the built-in processor GetTwitter is a pull. Are there streaming based processor for getting Tweets available? If no, any suggestions on how would l go by to build one?
>
> Thanks
> Jamie
GetTwitter to stream tweets instead pull
Posted by Jamie Wang <ja...@opentext.com>.
Hi,
I understand the built-in processor GetTwitter is a pull. Are there streaming based processor for getting Tweets available? If no, any suggestions on how would l go by to build one?
Thanks
Jamie
Re: MiNiFi's differentiator
Posted by Jeff Zemerick <jz...@apache.org>.
Thanks for that info, Aldrin! I will be glad to look over it.
Jeff
On Thu, Apr 20, 2017 at 12:09 PM, Aldrin Piri <al...@gmail.com> wrote:
> Hey Jeff,
>
> Just to hone in a bit more, MiNiFi would likely consume from some of the
> work surrounding the Command and Control [1] efforts.
>
> There is some initial work under way which should serve as a nice
> foundation to start exploring a more full featured and richer
> implementation. Certainly plan to further expand and update docs as we
> learn a bit more through design and seeing what supports the common needs.
> Would certainly invite you to evaluate how it would apply to your uses and
> comment, if it is of interest.
>
> Thanks!
>
> [1] https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control
>
> On Thu, Apr 20, 2017 at 11:51 AM, Jeff Zemerick <jz...@apache.org>
> wrote:
>
>> Hi Joe,
>>
>> No, the documentation is clear that is currently only available
>> implementation. Since it was designed to be extendable, I was wondering if
>> there were any example cases in which using the WholeConfigDifferentiator
>> might not be the ideal differentiator and I should provide my own
>> implementation. WholeConfigDifferentiator's implementation seems simple,
>> fast, and to the point. I wasn't aware of the NiFi-Registry project so that
>> helps me see the bigger picture.
>>
>> Thanks,
>> Jeff
>>
>>
>> On Thu, Apr 20, 2017 at 10:36 AM, Joe Percivall <jp...@apache.org>
>> wrote:
>>
>>> Hello Jeff,
>>>
>>> Glad to hear the WholeConfigDifferentiator is working well. While it is
>>> configurable, that is currently the only implementation. Is there a
>>> specific place that suggests there are currently more implementations? The
>>> documentation lists it as the only one[1]. When I wrote it up I implemented
>>> it in such a way so that we could easily add other differentiators in the
>>> future. One such example is once we have the NiFi-Registry in place, a
>>> differentiator that takes advantage of whatever the uniqueness scheme that
>>> is implemented for that.
>>>
>>> [1] https://github.com/apache/nifi-minifi/blob/master/minifi
>>> -docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
>>>
>>> Joe
>>>
>>> On Thu, Apr 20, 2017 at 10:06 AM, Jeff Zemerick <jz...@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> MiNiFi's WholeConfigDifferentiator along with
>>>> the PullHttpChangeIngestor is working well for me. I see that
>>>> the differentiator is configurable and additional implementations can be
>>>> provided. Are there any examples of circumstances in which the using
>>>> WholeConfigDifferentiator would not be the best choice?
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>
>>>
>>>
>>> --
>>> *Joe Percivall*
>>> linkedin.com/in/Percivall
>>> e: jpercivall@apache.com
>>>
>>
>>
>
Re: MiNiFi's differentiator
Posted by Aldrin Piri <al...@gmail.com>.
Hey Jeff,
Just to hone in a bit more, MiNiFi would likely consume from some of the
work surrounding the Command and Control [1] efforts.
There is some initial work under way which should serve as a nice
foundation to start exploring a more full featured and richer
implementation. Certainly plan to further expand and update docs as we
learn a bit more through design and seeing what supports the common needs.
Would certainly invite you to evaluate how it would apply to your uses and
comment, if it is of interest.
Thanks!
[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
On Thu, Apr 20, 2017 at 11:51 AM, Jeff Zemerick <jz...@apache.org>
wrote:
> Hi Joe,
>
> No, the documentation is clear that is currently only available
> implementation. Since it was designed to be extendable, I was wondering if
> there were any example cases in which using the WholeConfigDifferentiator might
> not be the ideal differentiator and I should provide my own implementation.
> WholeConfigDifferentiator's implementation seems simple, fast, and to the
> point. I wasn't aware of the NiFi-Registry project so that helps me see the
> bigger picture.
>
> Thanks,
> Jeff
>
>
> On Thu, Apr 20, 2017 at 10:36 AM, Joe Percivall <jp...@apache.org>
> wrote:
>
>> Hello Jeff,
>>
>> Glad to hear the WholeConfigDifferentiator is working well. While it is
>> configurable, that is currently the only implementation. Is there a
>> specific place that suggests there are currently more implementations? The
>> documentation lists it as the only one[1]. When I wrote it up I implemented
>> it in such a way so that we could easily add other differentiators in the
>> future. One such example is once we have the NiFi-Registry in place, a
>> differentiator that takes advantage of whatever the uniqueness scheme that
>> is implemented for that.
>>
>> [1] https://github.com/apache/nifi-minifi/blob/master/minifi
>> -docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
>>
>> Joe
>>
>> On Thu, Apr 20, 2017 at 10:06 AM, Jeff Zemerick <jz...@apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> MiNiFi's WholeConfigDifferentiator along with
>>> the PullHttpChangeIngestor is working well for me. I see that
>>> the differentiator is configurable and additional implementations can be
>>> provided. Are there any examples of circumstances in which the using
>>> WholeConfigDifferentiator would not be the best choice?
>>>
>>> Thanks,
>>> Jeff
>>>
>>
>>
>>
>> --
>> *Joe Percivall*
>> linkedin.com/in/Percivall
>> e: jpercivall@apache.com
>>
>
>
Re: MiNiFi's differentiator
Posted by Jeff Zemerick <jz...@apache.org>.
Hi Joe,
No, the documentation is clear that is currently only available
implementation. Since it was designed to be extendable, I was wondering if
there were any example cases in which using the WholeConfigDifferentiator might
not be the ideal differentiator and I should provide my own implementation.
WholeConfigDifferentiator's implementation seems simple, fast, and to the
point. I wasn't aware of the NiFi-Registry project so that helps me see the
bigger picture.
Thanks,
Jeff
On Thu, Apr 20, 2017 at 10:36 AM, Joe Percivall <jp...@apache.org>
wrote:
> Hello Jeff,
>
> Glad to hear the WholeConfigDifferentiator is working well. While it is
> configurable, that is currently the only implementation. Is there a
> specific place that suggests there are currently more implementations? The
> documentation lists it as the only one[1]. When I wrote it up I implemented
> it in such a way so that we could easily add other differentiators in the
> future. One such example is once we have the NiFi-Registry in place, a
> differentiator that takes advantage of whatever the uniqueness scheme that
> is implemented for that.
>
> [1] https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#
> automatic-warm-redeploy
>
> Joe
>
> On Thu, Apr 20, 2017 at 10:06 AM, Jeff Zemerick <jz...@apache.org>
> wrote:
>
>> Hi all,
>>
>> MiNiFi's WholeConfigDifferentiator along with the PullHttpChangeIngestor
>> is working well for me. I see that the differentiator is configurable and
>> additional implementations can be provided. Are there any examples of
>> circumstances in which the using WholeConfigDifferentiator would not be the
>> best choice?
>>
>> Thanks,
>> Jeff
>>
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jpercivall@apache.com
>
Re: MiNiFi's differentiator
Posted by Joe Percivall <jp...@apache.org>.
Hello Jeff,
Glad to hear the WholeConfigDifferentiator is working well. While it is
configurable, that is currently the only implementation. Is there a
specific place that suggests there are currently more implementations? The
documentation lists it as the only one[1]. When I wrote it up I implemented
it in such a way so that we could easily add other differentiators in the
future. One such example is once we have the NiFi-Registry in place, a
differentiator that takes advantage of whatever the uniqueness scheme that
is implemented for that.
[1]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
Joe
On Thu, Apr 20, 2017 at 10:06 AM, Jeff Zemerick <jz...@apache.org>
wrote:
> Hi all,
>
> MiNiFi's WholeConfigDifferentiator along with the PullHttpChangeIngestor
> is working well for me. I see that the differentiator is configurable and
> additional implementations can be provided. Are there any examples of
> circumstances in which the using WholeConfigDifferentiator would not be the
> best choice?
>
> Thanks,
> Jeff
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jpercivall@apache.com