You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by praveen S <my...@gmail.com> on 2016/01/14 08:44:27 UTC
Usage of SparkContext within a Web container
Is use of SparkContext from a Web container a right way to process spark
jobs or should we use spark-submit in a processbuilder?
Are there any pros or cons of using SparkContext from a Web container..?
How does zeppelin trigger spark jobs from the Web context?
Re: Usage of SparkContext within a Web container
Posted by Eugene Morozov <ev...@gmail.com>.
Praveen,
Zeppelin uses Spark's REPL.
I'm currently writing an app that is a web service, which is going to run
spark jobs.
So, at the init stage I just create JavaSparkContext and then use it for
all users requests. Web service is stateless. The issue with stateless is
that it's possible to run several instances of web service, but each of
them will have separate JavaSparkContext, which means they are going to
compete for resources as different application. Although they have to look
as just one application. I'm pretty sure it's possible to use pools, but I
haven't tried it, yet. I see no other cons... or pros for that matter.
The way you're going to use it, I'd say, depends on if users are going to
provide their own code. If that's the case, then you probably better with
Zeppelin's way. If not - then my assumption is that using SparkContext for
processing is simpler.
--
Be well!
Jean Morozov
On Thu, Jan 14, 2016 at 10:44 AM, praveen S <my...@gmail.com> wrote:
> Is use of SparkContext from a Web container a right way to process spark
> jobs or should we use spark-submit in a processbuilder?
>
> Are there any pros or cons of using SparkContext from a Web container..?
>
> How does zeppelin trigger spark jobs from the Web context?
>
Re: Usage of SparkContext within a Web container
Posted by Eugene Morozov <ev...@gmail.com>.
Praveen,
Zeppelin uses Spark's REPL.
I'm currently writing an app that is a web service, which is going to run
spark jobs.
So, at the init stage I just create JavaSparkContext and then use it for
all users requests. Web service is stateless. The issue with stateless is
that it's possible to run several instances of web service, but each of
them will have separate JavaSparkContext, which means they are going to
compete for resources as different application. Although they have to look
as just one application. I'm pretty sure it's possible to use pools, but I
haven't tried it, yet. I see no other cons... or pros for that matter.
The way you're going to use it, I'd say, depends on if users are going to
provide their own code. If that's the case, then you probably better with
Zeppelin's way. If not - then my assumption is that using SparkContext for
processing is simpler.
--
Be well!
Jean Morozov
On Thu, Jan 14, 2016 at 10:44 AM, praveen S <my...@gmail.com> wrote:
> Is use of SparkContext from a Web container a right way to process spark
> jobs or should we use spark-submit in a processbuilder?
>
> Are there any pros or cons of using SparkContext from a Web container..?
>
> How does zeppelin trigger spark jobs from the Web context?
>