Fetching data from a REST api into storm.

[Yogini Gulkotwar <yo...@flutura.com>]

Hello friends,

I am trying to fetch data from one of my tools which has a REST api
feature. So basically, I need to know if it is possible to make calls to
the web api of my tool and fetch data.

It would be great if someone could share some example code where something
similar has been achieved.



Thanks & Regards,
Yogini Gulkotwar│Data Scientist

Re: Fetching data from a REST api into storm.

[Michael Rose <mi...@fullcontact.com>]

Anything you can do in normal Java code, you can do inside a Storm bolt.
Check out any Java HTTP client (OkHttp, Apache HTTP Client, JerseyClient,
URLConnection etc.) and use it to fetch the data you need.

If your REST call takes a long time, it might be worth doing so
asynchronously and emitting the results downstream when the thread returns.
Just make sure that you're synchronizing your OutputCollector when emitting
from a multithreaded context.

Hope that helps.

Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com


On Thu, Jul 24, 2014 at 10:47 AM, Yogini Gulkotwar <
yogini.gulkotwar@flutura.com> wrote:

> Hello friends,
>
> I am trying to fetch data from one of my tools which has a REST api
> feature. So basically, I need to know if it is possible to make calls to
> the web api of my tool and fetch data.
>
> It would be great if someone could share some example code where something
> similar has been achieved.
>
>
>
> Thanks & Regards,
> Yogini Gulkotwar│Data Scientist
>
>

Re: Fetching data from a REST api into storm.

[Philip Shon <ph...@gmail.com>]

Was not 100% sure of what you were looking for, but I found this to be
invaluable when dealing with the REST API.
https://github.com/apache/incubator-storm/blob/master/STORM-UI-REST-API.md


On Thu, Jul 24, 2014 at 11:47 AM, Yogini Gulkotwar <
yogini.gulkotwar@flutura.com> wrote:

> Hello friends,
>
> I am trying to fetch data from one of my tools which has a REST api
> feature. So basically, I need to know if it is possible to make calls to
> the web api of my tool and fetch data.
>
> It would be great if someone could share some example code where something
> similar has been achieved.
>
>
>
> Thanks & Regards,
> Yogini Gulkotwar│Data Scientist
>
>

Re: Fetching data from a REST api into storm.

[Mark Mandel <ma...@gmail.com>]

Yep, it's possible, I'm doing it right now.

I'm doing it with Clojure + Trident, but the result should be the same.

We've found that calls to REST are our biggest bottleneck, so be careful
how you call them and/or how often.

Mark


On Fri, Jul 25, 2014 at 2:47 AM, Yogini Gulkotwar <
yogini.gulkotwar@flutura.com> wrote:

> Hello friends,
>
> I am trying to fetch data from one of my tools which has a REST api
> feature. So basically, I need to know if it is possible to make calls to
> the web api of my tool and fetch data.
>
> It would be great if someone could share some example code where something
> similar has been achieved.
>
>
>
> Thanks & Regards,
> Yogini Gulkotwar│Data Scientist
>
>


-- 
E: mark.mandel@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast
http://www.2ddu.com/