[jira] [Commented] (SPARK-2389) globally shared SparkContext / shared Spark "application"

["Ashish Rawat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703595#comment-14703595 ] 

Ashish Rawat commented on SPARK-2389:
-------------------------------------

Hi [~pwendell],

We are facing exactly the problem which you mentioned. And we are looking for exactly the same solution that you mentioned i.e. Driver HA :)

I have a few questions/comments on the perspective you shared:
1. "If we started to go down this path, we'd need to do things like define a standard serialization format for the RDD data, a global namespace for RDD's, persistence, etc. And then you're building a filesystem."
You only need to preserve the RDD metainfo and not the actual RDDs, so there should not be any complexity of serialization format for RDD data.
2. Although the cache data is recoverable, but how to reduce the latency of building back a cache of TBs, for a live application?
3. Can we not just prevent executors from shutting, preserve some important driver info and connect back the driver? Or provide a Hot standby for driver?

> globally shared SparkContext / shared Spark "application"
> ---------------------------------------------------------
>
>                 Key: SPARK-2389
>                 URL: https://issues.apache.org/jira/browse/SPARK-2389
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Robert Stupp
>
> The documentation (in Cluster Mode Overview) cites:
> bq. Each application gets its own executor processes, which *stay up for the duration of the whole application* and run tasks in multiple threads. This has the benefit of isolating applications from each other, on both the scheduling side (each driver schedules its own tasks) and executor side (tasks from different applications run in different JVMs). However, it also means that *data cannot be shared* across different Spark applications (instances of SparkContext) without writing it to an external storage system.
> IMO this is a limitation that should be lifted to support any number of --driver-- client processes to share executors and to share (persistent / cached) data.
> This is especially useful if you have a bunch of frontend servers (dump web app servers) that want to use Spark as a _big computing machine_. Most important is the fact that Spark is quite good in caching/persisting data in memory / on disk thus removing load from backend data stores.
> Means: it would be really great to let different --driver-- client JVMs operate on the same RDDs and benefit from Spark's caching/persistence.
> It would however introduce some administration mechanisms to
> * start a shared context
> * update the executor configuration (# of worker nodes, # of cpus, etc) on the fly
> * stop a shared context
> Even "conventional" batch MR applications would benefit if ran fequently against the same data set.
> As an implicit requirement, RDD persistence could get a TTL for its materialized state.
> With such a feature the overall performance of today's web applications could then be increased by adding more web app servers, more spark nodes, more nosql nodes etc



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org