You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ha...@tutanota.com on 2020/01/19 22:55:12 UTC
RESTful Operations
I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However searching Spark's website docs
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
https://spark.apache.org/docs/latest/
I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server.
Any places I should start if I want to use Spark for such purpose?
Thanks
Re: RESTful Operations
Posted by ha...@tutanota.com.
I see. Thanks for the advice!
Jan 20, 2020, 09:47 by hemant2184@gmail.com:
> Livy has both statement based(scala) as well as batch processing(code jar packaged). I think first statement based approach is what you might want to look at.
> Data has to residing in some source.
>
> Thanks,
> Hemant
>
> On Mon, 20 Jan 2020 at 2:04 PM, <> hamishberridge@tutanota.com> > wrote:
>
>> Sorry didn't explain well. Livy seems to me similar to job server by interacting through restful api interface for job submission. The task I want to achieve is more like general restful api interface where user would provide some parameters for business operations (a bit like readText or a custom date source with embedded http). So when the spark receive those requests in e.g custom date source, the spark job can then perform normal operations such as aggregation, window functions, and so on.
>>
>> Thank you for the suggestions and help.
>>
>>
>>
>>
>> Jan 20, 2020, 00:26 by >> chris.teoh@gmail.com>> :
>>
>>>
>>>
>>> Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.
>>>
>>> On Mon, 20 Jan 2020 at 06:55, <>>> hamishberridge@tutanota.com>>> > wrote:
>>>
>>>> I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However searching Spark's website docs
>>>>
>>>> https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
>>>>
>>>> >>>> https://spark.apache.org/docs/latest/
>>>>
>>>> I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server.
>>>>
>>>> Any places I should start if I want to use Spark for such purpose?
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>> --
>>> Chris
>>>
>>
>>
Re: RESTful Operations
Posted by hemant singh <he...@gmail.com>.
Livy has both statement based(scala) as well as batch processing(code jar
packaged). I think first statement based approach is what you might want to
look at.
Data has to residing in some source.
Thanks,
Hemant
On Mon, 20 Jan 2020 at 2:04 PM, <ha...@tutanota.com> wrote:
> Sorry didn't explain well. Livy seems to me similar to job server by
> interacting through restful api interface for job submission. The task I
> want to achieve is more like general restful api interface where user would
> provide some parameters for business operations (a bit like readText or a
> custom date source with embedded http). So when the spark receive those
> requests in e.g custom date source, the spark job can then perform normal
> operations such as aggregation, window functions, and so on.
>
> Thank you for the suggestions and help.
>
>
>
>
> Jan 20, 2020, 00:26 by chris.teoh@gmail.com:
>
> Maybe something like Livy, otherwise roll your own REST API and have it
> start a Spark job.
>
> On Mon, 20 Jan 2020 at 06:55, <ha...@tutanota.com> wrote:
>
> I am new to Spark. The task I want to accomplish is let client send http
> requests, then spark process that request for further operations. However
> searching Spark's website docs
>
>
> https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
> <https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package>
>
> https://spark.apache.org/docs/latest/
>
> I do not find any places mentioning about this. Also most of the internet
> result are more lated to spark job server.
>
> Any places I should start if I want to use Spark for such purpose?
>
> Thanks
>
>
>
> --
> Chris
>
>
>
Re: RESTful Operations
Posted by ha...@tutanota.com.
Sorry didn't explain well. Livy seems to me similar to job server by interacting through restful api interface for job submission. The task I want to achieve is more like general restful api interface where user would provide some parameters for business operations (a bit like readText or a custom date source with embedded http). So when the spark receive those requests in e.g custom date source, the spark job can then perform normal operations such as aggregation, window functions, and so on.
Thank you for the suggestions and help.
Jan 20, 2020, 00:26 by chris.teoh@gmail.com:
> Maybe something like Livy, otherwise roll your own REST API and have it start a Spark job.
>
> On Mon, 20 Jan 2020 at 06:55, <> hamishberridge@tutanota.com> > wrote:
>
>> I am new to Spark. The task I want to accomplish is let client send http requests, then spark process that request for further operations. However searching Spark's website docs
>>
>> https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
>>
>> >> https://spark.apache.org/docs/latest/
>>
>> I do not find any places mentioning about this. Also most of the internet result are more lated to spark job server.
>>
>> Any places I should start if I want to use Spark for such purpose?
>>
>> Thanks
>>
>
>
> --
> Chris
>
Re: RESTful Operations
Posted by Chris Teoh <ch...@gmail.com>.
Maybe something like Livy, otherwise roll your own REST API and have it
start a Spark job.
On Mon, 20 Jan 2020 at 06:55, <ha...@tutanota.com> wrote:
> I am new to Spark. The task I want to accomplish is let client send http
> requests, then spark process that request for further operations. However
> searching Spark's website docs
>
>
> https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package
> <https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package>
>
> https://spark.apache.org/docs/latest/
>
> I do not find any places mentioning about this. Also most of the internet
> result are more lated to spark job server.
>
> Any places I should start if I want to use Spark for such purpose?
>
> Thanks
>
--
Chris