Many Spark metric names do not include the application name

[Amit Sela <am...@gmail.com>]

Hi guys,


It seems that JvmSource / DAGSchedulerSource / BlockManagerSource
/ ExecutorAllocationManager and other metrics sources (except for the
StreamingSource) publish their metrics directly under the "driver" fragment
(or its executor counter-part) of the metric path without including the
application name.


For instance:

   - "spark.application_xxxx.driver.DAGScheduler.job.allJobs"
   - while I would expect it to be something like:


   - *"*spark.application_xxxx.driver*.myAppName.*DAGScheduler.job.allJobs
   *"*
   - just like it currently is in the *streaming* metrics
   (StreamingSource):


   - "spark.application_xxxx.driver.*myAppName*
   .StreamingMetrics.streaming.lastCompletedBatch_processingDelay"

I was wondering if there is a reason for not including the application name
in the metric path?


Your help would be much appreciated!


Regards,

Amit