[jira] [Created] (SPARK-3270) Spark API for Embedded Applications

["Michal Malohlava (JIRA)" <ji...@apache.org>]

Michal Malohlava created SPARK-3270:
---------------------------------------

             Summary: Spark API for Embedded Applications
                 Key: SPARK-3270
                 URL: https://issues.apache.org/jira/browse/SPARK-3270
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: Michal Malohlava


At the begining, let's clarify my motivation - I would like to extend Spark platform by an embedded application (e.g., monitoring network performance in the context of selected applications) which will be launched on particular nodes in cluster with their launch.
Nevertheless, I do not want to modify Spark code directly and hardcode my code in, but I would prefer to provide a jar which would be registered and launched by Spark itself. 

Hence, to support such 3rd party applications (aka "extesions"/"plugins") Spark platform should provide at least:
  - an API to register an extension 
  - an API to register a "service" (meaning provided functionality)
  - well-defined points in Spark infrastructure which can be enriched/hooked by extension
     - in master/worker lifecycle
     - in applications lifecycle
     - in RDDs lifecycle
     - monitoring/reporting
     - ...
  - a way of deploying extension (for example, simply putting the extension on classpath and using Java service interface)

In this context, I do not mean reinventing OSGi (or another plugin platform) but it can serve as a good starting point.





--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org