Stop spark application when the job is complete.

[Imre Nagi <im...@gmail.com>]

Hi,

I have a spark application for batch processing in standalone cluster. The
job is to query the database and then do some transformation, aggregation,
and several actions such as indexing the result into the elasticsearch.

If I dont call the sc.stop(), the spark application wont stop and take will
keep the resource used by the application. In the other hand, if I call the
sc.stop(), the spark app will be stopped before it query the database and
do further processing.

Can anyone help me to give best practice in stopping the spark application
when the job is complete?

Thanks,
Imre

Re: Stop spark application when the job is complete.

[Ted Yu <yu...@gmail.com>]

Can you call sc.stop() after indexing into elastic search ?

> On Mar 16, 2016, at 9:17 PM, Imre Nagi <im...@gmail.com> wrote:
> 
> Hi,
> 
> I have a spark application for batch processing in standalone cluster. The job is to query the database and then do some transformation, aggregation, and several actions such as indexing the result into the elasticsearch.
> 
> If I dont call the sc.stop(), the spark application wont stop and take will keep the resource used by the application. In the other hand, if I call the sc.stop(), the spark app will be stopped before it query the database and do further processing.
> 
> Can anyone help me to give best practice in stopping the spark application when the job is complete?
> 
> Thanks,
> Imre

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org