You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Minnow Noir <mi...@gmail.com> on 2015/03/31 22:46:46 UTC
Query REST web service with Spark?
We have have some data on Hadoop that needs augmented with data only
available to us via a REST service. We're using Spark to search for, and
correct, missing data. Even though there are a lot of records to scour for
missing data, the total number of calls to the service is expected to be
low, so it would be ideal to do the whole job in Spark as we scour the data.
I don't see anything obvious in the API or on Google relating to making
REST calls from a Spark job. Is it possible?
Thanks,
Alec
Re: Query REST web service with Spark?
Posted by Emre Sevinc <em...@gmail.com>.
Hello Minnow,
It is possible. You can for example use Jersey REST client to query a web
service and get its results in a Spark job. In fact, that's what we did
actually in a recent project (in a Spark Streaming application).
Kind regards,
Emre Sevinç
http://www.bigindustries.be/
On Tue, Mar 31, 2015 at 10:46 PM, Minnow Noir <mi...@gmail.com> wrote:
> We have have some data on Hadoop that needs augmented with data only
> available to us via a REST service. We're using Spark to search for, and
> correct, missing data. Even though there are a lot of records to scour for
> missing data, the total number of calls to the service is expected to be
> low, so it would be ideal to do the whole job in Spark as we scour the data.
>
> I don't see anything obvious in the API or on Google relating to making
> REST calls from a Spark job. Is it possible?
>
> Thanks,
>
> Alec
>
--
Emre Sevinc
Re: Query REST web service with Spark?
Posted by Todd Nist <ts...@gmail.com>.
Here are a few ways to achieve what your loolking to do:
https://github.com/cjnolet/spark-jetty-server
Spark Job Server - https://github.com/spark-jobserver/spark-jobserver -
defines a REST API for Spark
Hue -
http://gethue.com/get-started-with-spark-deploy-spark-server-and-compute-pi-from-your-web-browser/
Spark Kernel project: https://github.com/ibm-et/spark-kernel
> The Spark Kernel's goal is to serve as the foundation for interactive
> applications. The project provides a client library in Scala that abstracts
> connecting to the kernel (containing a Spark Context), which can be
> embedded into a web application. We demonstrated this at StataConf when we
> embedded the Spark Kernel client into a Play application to provide an
> interactive web application that communicates to Spark via the Spark Kernel
> (hosting a SparkContext).
Hopefully one of those will give you what your looking for.
-Todd
On Tue, Mar 31, 2015 at 5:06 PM, Burak Yavuz <br...@gmail.com> wrote:
> Hi,
>
> If I recall correctly, I've read people integrating REST calls to Spark
> Streaming jobs in the user list. I don't imagine any cases for why it
> shouldn't be possible.
>
> Best,
> Burak
>
> On Tue, Mar 31, 2015 at 1:46 PM, Minnow Noir <mi...@gmail.com> wrote:
>
>> We have have some data on Hadoop that needs augmented with data only
>> available to us via a REST service. We're using Spark to search for, and
>> correct, missing data. Even though there are a lot of records to scour for
>> missing data, the total number of calls to the service is expected to be
>> low, so it would be ideal to do the whole job in Spark as we scour the data.
>>
>> I don't see anything obvious in the API or on Google relating to making
>> REST calls from a Spark job. Is it possible?
>>
>> Thanks,
>>
>> Alec
>>
>
>
Re: Query REST web service with Spark?
Posted by Burak Yavuz <br...@gmail.com>.
Hi,
If I recall correctly, I've read people integrating REST calls to Spark
Streaming jobs in the user list. I don't imagine any cases for why it
shouldn't be possible.
Best,
Burak
On Tue, Mar 31, 2015 at 1:46 PM, Minnow Noir <mi...@gmail.com> wrote:
> We have have some data on Hadoop that needs augmented with data only
> available to us via a REST service. We're using Spark to search for, and
> correct, missing data. Even though there are a lot of records to scour for
> missing data, the total number of calls to the service is expected to be
> low, so it would be ideal to do the whole job in Spark as we scour the data.
>
> I don't see anything obvious in the API or on Google relating to making
> REST calls from a Spark job. Is it possible?
>
> Thanks,
>
> Alec
>
Re: Query REST web service with Spark?
Posted by Ashish Rangole <ar...@gmail.com>.
All you need is a client to the target REST service in your Spark task. It
could be as simple as a HttpClient. Most likely that client won't be
serializable in which case you initialize it lazily. There are useful
examples in Spark knowledge base gitbook that you can look at.
On Mar 31, 2015 1:48 PM, "Minnow Noir" <mi...@gmail.com> wrote:
We have have some data on Hadoop that needs augmented with data only
available to us via a REST service. We're using Spark to search for, and
correct, missing data. Even though there are a lot of records to scour for
missing data, the total number of calls to the service is expected to be
low, so it would be ideal to do the whole job in Spark as we scour the data.
I don't see anything obvious in the API or on Google relating to making
REST calls from a Spark job. Is it possible?
Thanks,
Alec