[jira] [Commented] (TOREE-390) Lazily start Spark sessions

["Ryan Blue (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990490#comment-15990490 ] 

Ryan Blue commented on TOREE-390:
---------------------------------

[~luciano resende], we aren't having the weird issues with requests any more, since the commit you linked to and [8f1775d|https://github.com/apache/incubator-toree/commit/8f1775d9dcdc56dd4f3a7c44abf0c36ae0417444].

The problem this fixes is the initial wait time is both startup for Toree and Spark. This separates the two waits, making it clear which one is due to Spark starting up and getting resources from YARN (in our deployment). This is better for the user experience because people don't think Toree is slow, they correctly attribute it to Spark. The secondary benefit is that users can customize their Spark session before it starts by creating one in the first cell, and Toree works just fine with that.

> Lazily start Spark sessions
> ---------------------------
>
>                 Key: TOREE-390
>                 URL: https://issues.apache.org/jira/browse/TOREE-390
>             Project: TOREE
>          Issue Type: Improvement
>            Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is taken by starting a Spark session and waiting for containers. Lazily starting Spark sessions helps the notebook environment feel faster, even if the user is waiting on Spark to start up because the time waiting for Spark to start is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that can't be changed in a Spark context. It also enables the same startup code that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
>     .config(...)
>     .setAppName(...)
>     .getOrCreate()
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)