Spark Metrics for Ganglia

[Khalid Latif <kh...@aalto.fi>]

I built Apache Spark on Ubuntu 14.04 LTS with the following command:

mvn -Pspark-ganglia-lgpl -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0 
-DskipTests clean package

Build was successful. Then, following modifications were made.

     1. Included "SPARK_LOCAL_IP=127.0.0.1" to the file 
$SPARK_HOME/conf/spark-env.sh to avoid the following warnings.

         16/04/27 17:45:54 WARN Utils: Your hostname, ganglia resolves 
to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface 
eth0)
         16/04/27 17:45:54 WARN Utils: Set SPARK_LOCAL_IP if you need to 
bind to another address

         After that, Spark started well without these warnings.

     2. To enable Ganglia metrics, following lines were included to the 
file $SPARK_HOME/conf/metrics.properties

         # Enable GangliaSink for all instances
*.sink.ganglia.class=org.apache.spark.metrics.sink.GangliaSink
         *.sink.ganglia.name=SparkCluster
         *.sink.ganglia.host=XYZ.XYZ.XYZ.XYZ (Replaced by real IP)
         *.sink.ganglia.port=8649
         *.sink.ganglia.period=10
         *.sink.ganglia.unit=seconds
         *.sink.ganglia.ttl=1
         *.sink.ganglia.mode=multicast

         Following errors were displayed, but Spark got started.

         16/04/27 17:45:59 ERROR MetricsSystem: Sink class 
org.apache.spark.metrics.sink.GangliaSink  cannot be instantiated
         16/04/27 17:45:59 ERROR SparkContext: Error initializing 
SparkContext.
         java.lang.ClassNotFoundException: 
org.apache.spark.metrics.sink.GangliaSink
             at 
scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
             at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
             at java.lang.Class.forName0(Native Method)
             .......


"GangliaSink" class can be found at: 
$SPARK_HOME/external/spark-ganglia-lgpl/target/classes/org/apache/spark/metrics/sink/GangliaSink.class

I can see previous threads regarding the same problem but I cannot find 
any solution. Any idea?