Custom metrics sink

[Christopher Piggott <cp...@gmail.com>]

Just for fun, i want to make a stupid program that makes different
frequency chimes as each worker becomes active.  That way you can 'hear'
what the cluster is doing and how it's distributing work.

I thought to do this I would make a custom Sink, but the Sink and
everything else in org.apache.spark.metrics.sink is private to spark.  What
I was hoping to do was to just pick up the # of active workers in semi real
time (once a second?) and have them send a UDP message somewhere... then
each worker would be assigned to a different frequency chime.  It's just a
toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?

--C

Re: Custom metrics sink

[Silvio Fiorito <si...@granturing.com>]

Just set your custom sink in the org.apache.spark.metrics.sink namespace and configure metrics.properties. Use ConsoleSink as an example. Obviously since it’s private the API may change, but in the meantime that should work…

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala#L28


From: Christopher Piggott <cp...@gmail.com>
Date: Friday, March 16, 2018 at 4:09 PM
To: "user@spark.apache.org" <us...@spark.apache.org>
Subject: Custom metrics sink

Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active.  That way you can 'hear' what the cluster is doing and how it's distributing work.

I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark.  What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime.  It's just a toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?

--C

Re: Custom metrics sink

[Felix Cheung <fe...@hotmail.com>]

There is a proposal to expose them. See SPARK-14151

________________________________
From: Christopher Piggott <cp...@gmail.com>
Sent: Friday, March 16, 2018 1:09:38 PM
To: user@spark.apache.org
Subject: Custom metrics sink

Just for fun, i want to make a stupid program that makes different frequency chimes as each worker becomes active.  That way you can 'hear' what the cluster is doing and how it's distributing work.

I thought to do this I would make a custom Sink, but the Sink and everything else in org.apache.spark.metrics.sink is private to spark.  What I was hoping to do was to just pick up the # of active workers in semi real time (once a second?) and have them send a UDP message somewhere... then each worker would be assigned to a different frequency chime.  It's just a toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?

--C