You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Bob Zhao <bo...@live.com> on 2015/12/09 02:24:14 UTC
The EndPoints of GetTwitter
Hello,
I want to know which provider is the Firehose Endpoint pointing to?
Or how could I get the 100% public content through NIFI?
Thanks,
Bob
RE: The EndPoints of GetTwitter
Posted by Bob Zhao <bo...@live.com>.
HI guys,
Thanks all for the reply. Forget Firehose now. Twitter plans to use Firehose by itself only.
http://blog.datasift.com/2015/04/11/twitter-ends-its-partnership-with-datasift-firehose-access-expires-on-august-13-2015/
http://siliconangle.com/blog/2015/04/13/twitter-cuts-off-firehose-access-eyes-big-data-bonanza/
Thanks,
Bob
> From: markap14@hotmail.com
> Subject: Re: The EndPoints of GetTwitter
> Date: Tue, 8 Dec 2015 21:17:38 -0500
> To: users@nifi.apache.org
>
> We use the Hosebird client provided by Twitter. It does expose a Firehose endpoint but of course your tokens would have to be validate for that endpoint. I've never actually tested it, since I don't have credentials for the Firehouse, but I would love to hear the results of using it. I believe it uses the gnip endpoint.
>
> Sent from my iPhone
>
> > On Dec 8, 2015, at 9:10 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > Is there any reason we'd not just let the user enter an override URL
> > to pull from?
> >
> >> On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao <bo...@live.com> wrote:
> >> Thanks for the quick response, Aldrin!
> >>
> >> I know we are not NSA. But there are at least 2 3rd parties who provide the
> >> Firehose stream(paid).
> >> Does NIFI have any plan to add their API later? After playing with the
> >> GetTwitter on the sample stream, I really like the way that NIFI handles
> >> these?
> >>
> >> Thanks,
> >> Bob
> >>
> >> ________________________________
> >> From: aldrinpiri@gmail.com
> >> Date: Tue, 8 Dec 2015 20:51:21 -0500
> >> Subject: Re: The EndPoints of GetTwitter
> >> To: users@nifi.apache.org
> >>
> >>
> >> Bob,
> >>
> >> The Firehose stream is not publicly available via Twitter and requires
> >> special authorization to utilize [1]. As a result, this endpoint was not
> >> incorporated into the GetTwitter processor's properties and there is no way
> >> to provide the totality of the public feed in NiFi.
> >>
> >> [1] https://dev.twitter.com/streaming/firehose
> >>
> >> On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
> >>
> >> Hello,
> >>
> >> I want to know which provider is the Firehose Endpoint pointing to?
> >> Or how could I get the 100% public content through NIFI?
> >>
> >>
> >> Thanks,
> >> Bob
> >>
> >>
Re: The EndPoints of GetTwitter
Posted by Mark Payne <ma...@hotmail.com>.
We use the Hosebird client provided by Twitter. It does expose a Firehose endpoint but of course your tokens would have to be validate for that endpoint. I've never actually tested it, since I don't have credentials for the Firehouse, but I would love to hear the results of using it. I believe it uses the gnip endpoint.
Sent from my iPhone
> On Dec 8, 2015, at 9:10 PM, Joe Witt <jo...@gmail.com> wrote:
>
> Is there any reason we'd not just let the user enter an override URL
> to pull from?
>
>> On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao <bo...@live.com> wrote:
>> Thanks for the quick response, Aldrin!
>>
>> I know we are not NSA. But there are at least 2 3rd parties who provide the
>> Firehose stream(paid).
>> Does NIFI have any plan to add their API later? After playing with the
>> GetTwitter on the sample stream, I really like the way that NIFI handles
>> these?
>>
>> Thanks,
>> Bob
>>
>> ________________________________
>> From: aldrinpiri@gmail.com
>> Date: Tue, 8 Dec 2015 20:51:21 -0500
>> Subject: Re: The EndPoints of GetTwitter
>> To: users@nifi.apache.org
>>
>>
>> Bob,
>>
>> The Firehose stream is not publicly available via Twitter and requires
>> special authorization to utilize [1]. As a result, this endpoint was not
>> incorporated into the GetTwitter processor's properties and there is no way
>> to provide the totality of the public feed in NiFi.
>>
>> [1] https://dev.twitter.com/streaming/firehose
>>
>> On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
>>
>> Hello,
>>
>> I want to know which provider is the Firehose Endpoint pointing to?
>> Or how could I get the 100% public content through NIFI?
>>
>>
>> Thanks,
>> Bob
>>
>>
Re: The EndPoints of GetTwitter
Posted by Aldrin Piri <al...@gmail.com>.
Main consideration is if the APIs aren't identical, this won't work out of
the box. At this point, it is not so much a GetTwitter as it is Get<Data
Provider> that has their own schema. Anecdotally, I have a bit more than a
foggy recollection that DataSift's API was accessed and possessed a
different "schema" than Twitter's HBC.
Code wise, there would be some work needed to integrate custom endpoints.
Again, this is in view of, and secondary to the previous consideration of
resource matching.
On Tue, Dec 8, 2015 at 9:10 PM, Joe Witt <jo...@gmail.com> wrote:
> Is there any reason we'd not just let the user enter an override URL
> to pull from?
>
> On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao <bo...@live.com> wrote:
> > Thanks for the quick response, Aldrin!
> >
> > I know we are not NSA. But there are at least 2 3rd parties who provide
> the
> > Firehose stream(paid).
> > Does NIFI have any plan to add their API later? After playing with the
> > GetTwitter on the sample stream, I really like the way that NIFI handles
> > these?
> >
> > Thanks,
> > Bob
> >
> > ________________________________
> > From: aldrinpiri@gmail.com
> > Date: Tue, 8 Dec 2015 20:51:21 -0500
> > Subject: Re: The EndPoints of GetTwitter
> > To: users@nifi.apache.org
> >
> >
> > Bob,
> >
> > The Firehose stream is not publicly available via Twitter and requires
> > special authorization to utilize [1]. As a result, this endpoint was not
> > incorporated into the GetTwitter processor's properties and there is no
> way
> > to provide the totality of the public feed in NiFi.
> >
> > [1] https://dev.twitter.com/streaming/firehose
> >
> > On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
> >
> > Hello,
> >
> > I want to know which provider is the Firehose Endpoint pointing to?
> > Or how could I get the 100% public content through NIFI?
> >
> >
> > Thanks,
> > Bob
> >
> >
>
Re: The EndPoints of GetTwitter
Posted by Joe Witt <jo...@gmail.com>.
Is there any reason we'd not just let the user enter an override URL
to pull from?
On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao <bo...@live.com> wrote:
> Thanks for the quick response, Aldrin!
>
> I know we are not NSA. But there are at least 2 3rd parties who provide the
> Firehose stream(paid).
> Does NIFI have any plan to add their API later? After playing with the
> GetTwitter on the sample stream, I really like the way that NIFI handles
> these?
>
> Thanks,
> Bob
>
> ________________________________
> From: aldrinpiri@gmail.com
> Date: Tue, 8 Dec 2015 20:51:21 -0500
> Subject: Re: The EndPoints of GetTwitter
> To: users@nifi.apache.org
>
>
> Bob,
>
> The Firehose stream is not publicly available via Twitter and requires
> special authorization to utilize [1]. As a result, this endpoint was not
> incorporated into the GetTwitter processor's properties and there is no way
> to provide the totality of the public feed in NiFi.
>
> [1] https://dev.twitter.com/streaming/firehose
>
> On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
>
> Hello,
>
> I want to know which provider is the Firehose Endpoint pointing to?
> Or how could I get the 100% public content through NIFI?
>
>
> Thanks,
> Bob
>
>
Re: The EndPoints of GetTwitter
Posted by Aldrin Piri <al...@gmail.com>.
I know that had been the case, but thought that Twitter had ceased or was
in the process of further curtailing that access.
Additionally, it seems I fibbed about the Firehose, sorry for the misstep
on that one. These endpoints are those exposed by the Hosebird Client that
Twitter provides in their library. GetTwitter effectively just wraps this
library in the NiFi framework.
In the case of 3rd parties, is the API the same as that which Twitter
publishes? What is the mechanism by which a client gets authenticated with
such resources?
Apologies again for misspeaking on the Firehose endpoint.
On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao <bo...@live.com> wrote:
> Thanks for the quick response, Aldrin!
>
> I know we are not NSA. But there are at least 2 3rd parties who provide
> the Firehose stream(paid).
> Does NIFI have any plan to add their API later? After playing with the
> GetTwitter on the sample stream, I really like the way that NIFI handles
> these?
>
> Thanks,
> Bob
>
> ------------------------------
> From: aldrinpiri@gmail.com
> Date: Tue, 8 Dec 2015 20:51:21 -0500
> Subject: Re: The EndPoints of GetTwitter
> To: users@nifi.apache.org
>
>
> Bob,
>
> The Firehose stream is not publicly available via Twitter and requires
> special authorization to utilize [1]. As a result, this endpoint was not
> incorporated into the GetTwitter processor's properties and there is no way
> to provide the totality of the public feed in NiFi.
>
> [1] https://dev.twitter.com/streaming/firehose
>
> On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
>
> Hello,
>
> I want to know which provider is the Firehose Endpoint pointing to?
> Or how could I get the 100% public content through NIFI?
>
>
> Thanks,
> Bob
>
>
>
RE: The EndPoints of GetTwitter
Posted by Bob Zhao <bo...@live.com>.
Thanks for the quick response, Aldrin!
I know we are not NSA. But there are at least 2 3rd parties who provide the Firehose stream(paid).
Does NIFI have any plan to add their API later? After playing with the GetTwitter on the sample stream, I really like the way that NIFI handles these?
Thanks,
Bob
From: aldrinpiri@gmail.com
Date: Tue, 8 Dec 2015 20:51:21 -0500
Subject: Re: The EndPoints of GetTwitter
To: users@nifi.apache.org
Bob,
The Firehose stream is not publicly available via Twitter and requires special authorization to utilize [1]. As a result, this endpoint was not incorporated into the GetTwitter processor's properties and there is no way to provide the totality of the public feed in NiFi.
[1] https://dev.twitter.com/streaming/firehose
On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
Hello,
I want to know which provider is the Firehose Endpoint pointing to?
Or how could I get the 100% public content through NIFI?
Thanks,
Bob
Re: The EndPoints of GetTwitter
Posted by Aldrin Piri <al...@gmail.com>.
Bob,
The Firehose stream is not publicly available via Twitter and requires
special authorization to utilize [1]. As a result, this endpoint was not
incorporated into the GetTwitter processor's properties and there is no way
to provide the totality of the public feed in NiFi.
[1] https://dev.twitter.com/streaming/firehose
On Tue, Dec 8, 2015 at 8:24 PM, Bob Zhao <bo...@live.com> wrote:
> Hello,
>
> I want to know which provider is the Firehose Endpoint pointing to?
> Or how could I get the 100% public content through NIFI?
>
>
> Thanks,
> Bob
>
>