You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "oskarryn (Jira)" <ji...@apache.org> on 2022/06/02 08:13:00 UTC
[jira] [Commented] (SPARK-33320) ExecutorMetrics are not written to CSV and StatsD sinks
[ https://issues.apache.org/jira/browse/SPARK-33320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17545339#comment-17545339 ]
oskarryn commented on SPARK-33320:
----------------------------------
I have the same issue while on Spark 2.4.8. Supposedly executors forward metrics to the driver and we should see ExecutorMetrics namespace in the driver component metrics, but sadly no executor metrics are available there.
> ExecutorMetrics are not written to CSV and StatsD sinks
> -------------------------------------------------------
>
> Key: SPARK-33320
> URL: https://issues.apache.org/jira/browse/SPARK-33320
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Environment: I was using Spark 2.4.4 on EMR with YARN. The relevant part of the config is below:
> {noformat}
> spark.metrics.executorMetricsSource.enabled=true
> spark.eventLog.logStageExecutorMetrics=true
> spark.metrics.conf.*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
> spark.metrics.conf.*.sink.servlet.class=org.apache.spark.metrics.sink.MetricsServlet
> spark.metrics.conf.*.sink.servlet.path=/home/hadoop/metrics/json
> spark.metrics.conf.*.sink.statsd.class=org.apache.spark.metrics.sink.StatsdSink
> spark.metrics.conf.*.sink.statsd.host=localhost
> spark.metrics.conf.*.sink.statsd.port=8125
> spark.metrics.conf.*.sink.statsd.period=10
> spark.metrics.conf.*.sink.statsd.unit=seconds
> spark.metrics.conf.*.sink.statsd.prefix=spark
> master.sink.servlet.path=/home/hadoop/metrics/master/json
> applications.sink.servlet.path=/home/hadoop/metrics/applications/json
> {noformat}
> Reporter: Peter Podlovics
> Priority: Major
>
> Metrics from the {{ExecutorMetrics}} namespace are not written to the CSV and StatsD sinks, even though some of them is available through the REST API (e.g.: {{memoryMetrics.usedOnHeapStorageMemory}}).
> I couldn't find the {{ExecutorMetrics}} either on the driver or the workers.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org