You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by unk1102 <um...@gmail.com> on 2018/09/16 18:15:06 UTC
Best practices on how to multiple spark sessions
Hi I have application which servers as ETL job and I have hundreds of such
ETL jobs which runs daily now as of now I have just one spark session which
is shared by all these jobs and sometimes all of these jobs run at the same
time causing spark session to die due memory issues mostly. Is this a good
design? I am thinking to create multiple spark sessions possibly one spark
session for each ETL job but there is delay in starting spark session which
seems to multiple by no of ETL jobs. Please share best practices and designs
for such problems. Thanks in advance.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Best practices on how to multiple spark sessions
Posted by Venkat Ramakrishnan <ve...@gmail.com>.
Umesh,
I found the following write-up dealing with architecture and memory
considerations elaborately. There are updates on memory, but it would
be a good start for you:
https://0x0fff.com/spark-architecture/
Any additional source(s) of info. are welcome from others too.
- Venkat.
On Sun, Sep 16, 2018 at 11:45 PM unk1102 <um...@gmail.com> wrote:
> Hi I have application which servers as ETL job and I have hundreds of such
> ETL jobs which runs daily now as of now I have just one spark session which
> is shared by all these jobs and sometimes all of these jobs run at the same
> time causing spark session to die due memory issues mostly. Is this a good
> design? I am thinking to create multiple spark sessions possibly one spark
> session for each ETL job but there is delay in starting spark session which
> seems to multiple by no of ETL jobs. Please share best practices and
> designs
> for such problems. Thanks in advance.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>