You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:04:27 UTC
[jira] [Updated] (SPARK-18115) Custom metrics Sink/Source prevent
Executor from starting
[ https://issues.apache.org/jira/browse/SPARK-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-18115:
---------------------------------
Labels: bulk-closed (was: )
> Custom metrics Sink/Source prevent Executor from starting
> ---------------------------------------------------------
>
> Key: SPARK-18115
> URL: https://issues.apache.org/jira/browse/SPARK-18115
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.0
> Reporter: Kostya Golikov
> Priority: Major
> Labels: bulk-closed
>
> Even though there is a semi-official support for custom metrics, in practice specifying either custom sink or custom source will lead to NoClassDefFound exceptions on executor side (but will be fine on driver side).
> The initialization goes as:
> 1. CoarseGrainedExecutorBackend [prepares SparkEnv for executor|https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L223]
> 2. SparkEnv [initializes MetricSystem|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala#L338-L351]. In case of executor it also starts it
> 3. On [`.start()` MetricsSystem parses configuration files and creates instances of sinks and sources|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L101-L102]. This is where the issue actually happens -- it tries to instantiate classes which are not there yet -- [jars and files are downloaded downstream, in Executor|https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/core/src/main/scala/org/apache/spark/executor/Executor.scala#L257]
> One of the possible solutions is to NOT start MetricSystem this fast, just like driver does, but postpone it until jar with user defined code is fetched and available on classpath.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org