You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Connie Yang <cy...@gmail.com> on 2014/09/08 08:51:56 UTC
Is there a Tweeter Streaming Spout?
Hi,
Is there a spout that stream tweeter feed based a list of hashtags?
Thanks,
Connie
Re: Is there a Tweeter Streaming Spout?
Posted by Harsha <st...@harsha.io>.
Just to note its an example spout . But I am not sure why it
wouldn't allow parallelism of more than 1. For twitter api its
http calls thats by increasing the parallelism of the spout you
are making more calls to twitter api. I think twitter rate
limits based on appid thats the limitation I can see but thats
not related to spout parallelism. Even with a single spout
instance you can go over your api call rate limit.
-Harsha
On Mon, Sep 8, 2014, at 06:35 AM, Vikas Agarwal wrote:
That is interesting. However, it won't allow spout parallelism
more than 1, right?
On Mon, Sep 8, 2014 at 6:56 PM, Harsha <[1...@harsha.io>
wrote:
Hi Connie,
You can take a look at twittersamplespout in
examples [2]https://github.com/apache/incubator-storm/blob/mast
er/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSa
mpleSpout.java
It uses twitter4j to read the api you can make changes to fit
your needs.
-Harsha
On Mon, Sep 8, 2014, at 12:15 AM, Vikas Agarwal wrote:
I guess no and it won't make sense to have one because it would
limit the parallelism of spout. Twitter stream allows only
single connection to the stream. You can use threading to have
parallelism in stream consumption but it would be difficult to
manage it with spouts. Better solution would be to writer
standalone twitter stream listener with multithreading and push
messages to Kafka (or some JMS queue) and then consume them
using KafkaSpout for instance. It would allow you to increase
parallelism of spout by the number of partitions of the topics.
On Mon, Sep 8, 2014 at 12:21 PM, Connie Yang
<[3...@gmail.com> wrote:
Hi,
Is there a spout that stream tweeter feed based a list of
hashtags?
Thanks,
Connie
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
[4]http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
[5]+1 (408) 988-2000 Work
[6]+1 (408) 716-2726 Fax
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
[7]http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
References
1. mailto:storm@harsha.io
2. https://github.com/apache/incubator-storm/blob/master/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSampleSpout.java
3. mailto:cyberconnie@gmail.com
4. http://www.infoobjects.com/
5. tel:%2B1%20%28408%29%20988-2000
6. tel:%2B1%20%28408%29%20716-2726
7. http://www.infoobjects.com/
Re: Is there a Tweeter Streaming Spout?
Posted by Vikas Agarwal <vi...@infoobjects.com>.
That is interesting. However, it won't allow spout parallelism more than 1,
right?
On Mon, Sep 8, 2014 at 6:56 PM, Harsha <st...@harsha.io> wrote:
> Hi Connie,
> You can take a look at twittersamplespout in examples
> https://github.com/apache/incubator-storm/blob/master/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSampleSpout.java
> It uses twitter4j to read the api you can make changes to fit your needs.
> -Harsha
>
>
> On Mon, Sep 8, 2014, at 12:15 AM, Vikas Agarwal wrote:
>
> I guess no and it won't make sense to have one because it would limit the
> parallelism of spout. Twitter stream allows only single connection to the
> stream. You can use threading to have parallelism in stream consumption but
> it would be difficult to manage it with spouts. Better solution would be to
> writer standalone twitter stream listener with multithreading and push
> messages to Kafka (or some JMS queue) and then consume them using
> KafkaSpout for instance. It would allow you to increase parallelism of
> spout by the number of partitions of the topics.
>
>
> On Mon, Sep 8, 2014 at 12:21 PM, Connie Yang <cy...@gmail.com>
> wrote:
>
> Hi,
>
> Is there a spout that stream tweeter feed based a list of hashtags?
>
> Thanks,
> Connie
>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Is there a Tweeter Streaming Spout?
Posted by Harsha <st...@harsha.io>.
Hi Connie,
You can take a look at twittersamplespout in
examples [1]https://github.com/apache/incubator-storm/blob/mast
er/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSa
mpleSpout.java
It uses twitter4j to read the api you can make changes to fit
your needs.
-Harsha
On Mon, Sep 8, 2014, at 12:15 AM, Vikas Agarwal wrote:
I guess no and it won't make sense to have one because it would
limit the parallelism of spout. Twitter stream allows only
single connection to the stream. You can use threading to have
parallelism in stream consumption but it would be difficult to
manage it with spouts. Better solution would be to writer
standalone twitter stream listener with multithreading and push
messages to Kafka (or some JMS queue) and then consume them
using KafkaSpout for instance. It would allow you to increase
parallelism of spout by the number of partitions of the topics.
On Mon, Sep 8, 2014 at 12:21 PM, Connie Yang
<[2...@gmail.com> wrote:
Hi,
Is there a spout that stream tweeter feed based a list of
hashtags?
Thanks,
Connie
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
[3]http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
References
1. https://github.com/apache/incubator-storm/blob/master/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSampleSpout.java
2. mailto:cyberconnie@gmail.com
3. http://www.infoobjects.com/
Re: Is there a Tweeter Streaming Spout?
Posted by Vikas Agarwal <vi...@infoobjects.com>.
I guess no and it won't make sense to have one because it would limit the
parallelism of spout. Twitter stream allows only single connection to the
stream. You can use threading to have parallelism in stream consumption but
it would be difficult to manage it with spouts. Better solution would be to
writer standalone twitter stream listener with multithreading and push
messages to Kafka (or some JMS queue) and then consume them using
KafkaSpout for instance. It would allow you to increase parallelism of
spout by the number of partitions of the topics.
On Mon, Sep 8, 2014 at 12:21 PM, Connie Yang <cy...@gmail.com> wrote:
> Hi,
>
> Is there a spout that stream tweeter feed based a list of hashtags?
>
> Thanks,
> Connie
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax