Monitoring with elastic search in spark job

[Xinyu Luan <xi...@moesif.com.INVALID>]

Hi, I am working on getting elastic search metrics(bytes_sent, bytes_accepted and bytes_retries) whenever we read/write data from elastic search in spark job. 

We tried to use spark listener, but still did get 0 byte by creating a listener class in the following attachment.

We. Also tried to use the recordsRead, recordsWritten, bytesRead and bytesWritten from each task to monitoring the data between rdd and elastic search. But got 0 after sending data to elastic search and calling the listener in spark job

```
recordsRead.addAndGet(taskEnd.taskMetrics.inputMetrics.recordsRead)
recordsWritten.addAndGet(taskEnd.taskMetrics.outputMetrics.recordsWritten)
bytesRead.addAndGet(taskEnd.taskMetrics.inputMetrics.bytesRead)
bytesWritten.addAndGet(taskEnd.taskMetrics.outputMetrics.bytesWritten)
```

We also tried to get the executor metrics (more details https://spark.apache.org/docs/latest/monitoring.html#executor-metrics <https://spark.apache.org/docs/latest/monitoring.html#executor-metrics>), but the route http://localhost:4040/api/v1/applications/[app-id]/executors <http://localhost:4040/api/v1/applications/%5Bapp-id%5D/executors>, does not work for us with the following setting.

.set("spark.ui.prometheus.enabled", "true")
.set("spark.eventLog.logStageExecutorMetrics", "true")

Can I get any suggestion or some examples for how to get the metrics correctly. 
Thank you for any help in advance.

Re: Monitoring with elastic search in spark job

[wilson <in...@bigcount.xyz>]

Maybe you can give a look at this?
https://github.com/banzaicloud/spark-metrics

regards


Xinyu Luan wrote:
> Can I get any suggestion or some examples for how to get the metrics 
> correctly.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org