You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Paul Schooss <pa...@gmail.com> on 2014/04/16 03:08:35 UTC
JMX with Spark
Has anyone got this working? I have enabled the properties for it in the
metrics.conf file and ensure that it is placed under spark's home
directory. Any ideas why I don't see spark beans ?
Re: JMX with Spark
Posted by Paul Schooss <pa...@gmail.com>.
Hello Folks,
Sorry for the delay, these emails got missed due to the volume.
Here is my metrics.conf
root@jobs-ab-hdn4:~# cat /opt/klout/spark/conf/metrics.conf
# syntax: [instance].sink|source.[name].[options]=[value]
# This file configures Spark's internal metrics system. The metrics system
is
# divided into instances which correspond to internal components.
# Each instance can be configured to report its metrics to one or more
sinks.
# Accepted values for [instance] are "master", "worker", "executor",
"driver",
# and "applications". A wild card "*" can be used as an instance name, in
# which case all instances will inherit the supplied property.
#
# Within an instance, a "source" specifies a particular set of grouped
metrics.
# there are two kinds of sources:
# 1. Spark internal sources, like MasterSource, WorkerSource, etc, which
will
# collect a Spark component's internal state. Each instance is paired
with a
# Spark source that is added automatically.
# 2. Common sources, like JvmSource, which will collect low level state.
# These can be added through configuration options and are then loaded
# using reflection.
#
# A "sink" specifies where metrics are delivered to. Each instance can be
# assigned one or more sinks.
#
# The sink|source field specifies whether the property relates to a sink or
# source.
#
# The [name] field specifies the name of source or sink.
#
# The [options] field is the specific property of this source or sink. The
# source or sink is responsible for parsing this property.
#
# Notes:
# 1. To add a new sink, set the "class" option to a fully qualified class
# name (see examples below).
# 2. Some sinks involve a polling period. The minimum allowed polling
period
# is 1 second.
# 3. Wild card properties can be overridden by more specific properties.
# For example, master.sink.console.period takes precedence over
# *.sink.console.period.
# 4. A metrics specific configuration
# "spark.metrics.conf=${SPARK_HOME}/conf/metrics.properties" should be
# added to Java properties using -Dspark.metrics.conf=xxx if you want to
# customize metrics system. You can also put the file in
${SPARK_HOME}/conf
# and it will be loaded automatically.
# 5. MetricsServlet is added by default as a sink in master, worker and
client
# driver, you can send http request "/metrics/json" to get a snapshot of
all the
# registered metrics in json format. For master, requests
"/metrics/master/json" and
# "/metrics/applications/json" can be sent seperately to get metrics
snapshot of
# instance master and applications. MetricsServlet may not be configured
by self.
#
## List of available sinks and their properties.
# org.apache.spark.metrics.sink.ConsoleSink
# Name: Default: Description:
# period 10 Poll period
# unit seconds Units of poll period
# org.apache.spark.metrics.sink.CSVSink
# Name: Default: Description:
# period 10 Poll period
# unit seconds Units of poll period
# directory /tmp Where to store CSV files
# org.apache.spark.metrics.sink.GangliaSink
# Name: Default: Description:
# host NONE Hostname or multicast group of Ganglia server
# port NONE Port of Ganglia server(s)
# period 10 Poll period
# unit seconds Units of poll period
# ttl 1 TTL of messages sent by Ganglia
# mode multicast Ganglia network mode ('unicast' or 'mulitcast')
#org.apache.spark.metrics.sink.JmxSink
# org.apache.spark.metrics.sink.MetricsServlet
# Name: Default: Description:
# path VARIES* Path prefix from the web server root
# sample false Whether to show entire set of samples for
histograms ('false' or 'true')
#
# * Default path is /metrics/json for all instances except the master. The
master has two paths:
# /metrics/aplications/json # App information
# /metrics/master/json # Master information
# org.apache.spark.metrics.sink.GraphiteSink
# Name: Default: Description:
# host NONE Hostname of Graphite server
# port NONE Port of Graphite server
# period 10 Poll period
# unit seconds Units of poll period
# prefix EMPTY STRING Prefix to prepend to metric name
## Examples
# Enable JmxSink for all instances by class name
*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
# Enable ConsoleSink for all instances by class name
#*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink
# Polling period for ConsoleSink
#*.sink.console.period=10
#*.sink.console.unit=seconds
# Master instance overlap polling period
#master.sink.console.period=15
#master.sink.console.unit=seconds
# Enable CsvSink for all instances
#*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
# Polling period for CsvSink
#*.sink.csv.period=1
#*.sink.csv.unit=minutes
# Polling directory for CsvSink
#*.sink.csv.directory=/tmp/
# Worker instance overlap polling period
#worker.sink.csv.period=10
#worker.sink.csv.unit=minutes
# Enable jvm source for instance master, worker, driver and executor
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
On Fri, Apr 25, 2014 at 4:20 AM, Ravi Hemnani <ra...@gmail.com>wrote:
> Can you share your working metrics.properties.?
>
> I want remote jmx to be enabled so i need to use the JMXSink and monitor my
> spark master and workers.
>
> But what are the parameters that are to be defined like host and port ?
>
> So your config can help.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/JMX-with-Spark-tp4309p4823.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
RE: JMX with Spark
Posted by Ravi Hemnani <ra...@gmail.com>.
Can you share your working metrics.properties.?
I want remote jmx to be enabled so i need to use the JMXSink and monitor my
spark master and workers.
But what are the parameters that are to be defined like host and port ?
So your config can help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JMX-with-Spark-tp4309p4823.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
RE: JMX with Spark
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Paul, would you please paste your metrics.conf out so that we can find the problems if you still have problems.
Thanks
Jerry
From: Parviz Deyhim [mailto:pdeyhim@gmail.com]
Sent: Wednesday, April 16, 2014 9:10 AM
To: user@spark.apache.org
Subject: Re: JMX with Spark
home directory or $home/conf directory? works for me with metrics.properties hosted under conf dir.
On Tue, Apr 15, 2014 at 6:08 PM, Paul Schooss <pa...@gmail.com>> wrote:
Has anyone got this working? I have enabled the properties for it in the metrics.conf file and ensure that it is placed under spark's home directory. Any ideas why I don't see spark beans ?
Re: JMX with Spark
Posted by Parviz Deyhim <pd...@gmail.com>.
home directory or $home/conf directory? works for me with
metrics.properties hosted under conf dir.
On Tue, Apr 15, 2014 at 6:08 PM, Paul Schooss <pa...@gmail.com>wrote:
> Has anyone got this working? I have enabled the properties for it in the
> metrics.conf file and ensure that it is placed under spark's home
> directory. Any ideas why I don't see spark beans ?
>