[jira] [Commented] (SPARK-3270) Spark API for Embedded Applications

["Xiangrui Meng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118332#comment-14118332 ] 

Xiangrui Meng commented on SPARK-3270:
--------------------------------------

[~mmalohlava] This is very useful. We should think about the common requirements of PySpark, Sparkling Water, and SparkR, and make them pluggable.

> Spark API for Embedded Applications
> -----------------------------------
>
>                 Key: SPARK-3270
>                 URL: https://issues.apache.org/jira/browse/SPARK-3270
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Michal Malohlava
>
> At the begining, let's clarify my motivation - I would like to extend Spark platform by an embedded application (e.g., monitoring network performance in the context of selected applications) which will be launched on particular nodes in cluster with their launch.
> Nevertheless, I do not want to modify Spark code directly and hardcode my code in, but I would prefer to provide a jar which would be registered and launched by Spark itself. 
> Hence, to support such 3rd party applications (aka "extesions"/"plugins") Spark platform should provide at least:
>   - an API to register an extension 
>   - an API to register a "service" (meaning provided functionality)
>   - well-defined points in Spark infrastructure which can be enriched/hooked by extension
>      - in master/worker lifecycle
>      - in applications lifecycle
>      - in RDDs lifecycle
>      - monitoring/reporting
>      - ...
>   - a way of deploying extension (for example, simply putting the extension on classpath and using Java service interface)
> In this context, I do not mean reinventing OSGi (or another plugin platform) but it can serve as a good starting point.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org