[jira] [Created] (SPARK-18115) Custom metrics Sink/Source prevent Executor from starting

["Kostya Golikov (JIRA)" <ji...@apache.org>]

Kostya Golikov created SPARK-18115:
--------------------------------------

             Summary: Custom metrics Sink/Source prevent Executor from starting
                 Key: SPARK-18115
                 URL: https://issues.apache.org/jira/browse/SPARK-18115
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.6.0
            Reporter: Kostya Golikov


Even though there is a semi-official support for custom metrics, in practice specifying either custom sink or custom source will lead to NoClassDefFound exceptions on executor side (but will be fine on driver side).

The initialization goes as: 

1. CoarseGrainedExecutorBackend [prepares SparkEnv for executor|https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L223]
2. SparkEnv [initializes MetricSystem|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala#L338-L351]. In case of executor it also starts it
3. On [`.start()` MetricsSystem parses configuration files and creates instances of sinks and sources|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L101-L102]. This is where the issue actually happens -- it tries to instantiate classes which are not there yet -- [jars and files are downloaded downstream, in Executor|https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/core/src/main/scala/org/apache/spark/executor/Executor.scala#L257]

One of the possible solutions is to NOT start MetricSystem this fast, just like driver does, but postpone it until jar with user defined code is fetched and available on classpath. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org