Using sparkContext.stop()

[Bruno Faria <br...@hotmail.com>]

Hey all,

I have created a Spark Job that runs successfully but if I do not use sc.stop() at the end, the job hangs. It shows some "cleaned accumulator 0" messages but never finishes.

I intent to use these jobs in production via spark-submit and schedule it in cron.

Is that the best practice use sc.stop() or is there something else I am missing.

One interesting point is, if I run the job for 100 lines, the job finishes completely (without using sc.stop(), but when running with actual data (more than millions) that happens.

i've waited for more than 24 hours but it never releases the prompt and in the UI it appears as RUNNING.

Appreciate any help

Thanks

Re: Using sparkContext.stop()

[Mich Talebzadeh <mi...@gmail.com>]

Hi

We are talking about spark streaming in here?

Depending on what is streamed, you can work out an exit strategy through
the total messages streamed in or through a time window in which you can
monitor the duration and exit if the duration > Window allocated (not to be
confused with windows interval etc).

HTH

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 9 September 2016 at 18:45, Bruno Faria <br...@hotmail.com> wrote:

> Hey all,
>
> I have created a Spark Job that runs successfully but if I do not use
> sc.stop() at the end, the job hangs. It shows some "cleaned accumulator 0"
> messages but never finishes.
>
> I intent to use these jobs in production via spark-submit and schedule it
> in cron.
>
> Is that the best practice use sc.stop() or is there something else I am
> missing.
>
> One interesting point is, if I run the job for 100 lines, the job finishes
> completely (without using sc.stop(), but when running with actual data
> (more than millions) that happens.
>
> i've waited for more than 24 hours but it never releases the prompt and in
> the UI it appears as RUNNING.
>
> Appreciate any help
>
> Thanks
>
>