You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Walter Underwood <wu...@wunderwood.org> on 2013/03/29 19:07:00 UTC

Solr metrics in Codahale metrics and Graphite?

What are folks using for this?

wunder
--
Walter Underwood
wunder@wunderwood.org

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Greg Pendlebury <gr...@gmail.com>.

Oh my bad. I thought it was already in. Thanks for the correction.

Ta,
Greg


On 17 March 2014 15:55, Shalin Shekhar Mangar <sh...@gmail.com>wrote:

> Greg, SOLR-4735 (using the codahale metrics lib) hasn't been committed
> yet. It is still work in progress.
>
> Actually the internal Solr Metrics class has a method to return 1
> minute stats but it is not used.
>
> On Mon, Mar 17, 2014 at 10:06 AM, Greg Pendlebury
> <gr...@gmail.com> wrote:
> > In the codahale metrics library there are 1, 5 and 15 minute moving
> > averages just like you would see in a tool like 'top'. However in Solr I
> > can only see 5 and 15 minute values, plus 'avgRequestsPerSecond'. I
> assumed
> > this was the 1 minute value initially, but it seems to be something like
> > the average since startup. I haven't looked thoroughly, but it is around
> 1%
> > of the other two in a normally idle test cluster after load tests have
> been
> > running for long enough that the 5 and 15 minute numbers match the load
> > testing throughput.
> >
> > Is this difference deliberate? or an accident? or am I wrong entirely? I
> > can compute the overall average anyway, given that the stats also include
> > the start time of the search handler and the total search count, so I
> > thought it might be an accident.
> >
> > Ta,
> > Greg
> >
> >
> >
> >
> >
> > On 4 May 2013 01:19, Furkan KAMACI <fu...@gmail.com> wrote:
> >
> >> Does anybody tested Ganglia with JMXTrans at production environment for
> >> SolrCloud?
> >>
> >> 2013/4/26 Dmitry Kan <so...@gmail.com>
> >>
> >> > Alan, Shawn,
> >> >
> >> > If backporting to 3.x is hard, no worries, we don't necessarily
> require
> >> the
> >> > patch as we are heading to 4.x eventually. It is just much easier
> within
> >> > our organization to test on the existing solr 3.4 as there are a few
> of
> >> > internal dependencies and custom code on top of solr. Also solr
> upgrades
> >> on
> >> > production systems are usually pushed forward by a month or so
> starting
> >> the
> >> > upgrade on development systems (requires lots of testing and
> >> > verifications).
> >> >
> >> > Nevertheless, it is good effort to make #solr #graphite friendly, so
> keep
> >> > it up! :)
> >> >
> >> > Dmitry
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Apr 25, 2013 at 9:29 PM, Shawn Heisey <so...@elyograg.org>
> wrote:
> >> >
> >> > > On 4/25/2013 6:30 AM, Dmitry Kan wrote:
> >> > > > We are very much interested in 3.4.
> >> > > >
> >> > > > On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk>
> >> > wrote:
> >> > > >> This is on top of trunk at the moment, but would be back ported
> to
> >> 4.4
> >> > > if
> >> > > >> there was interest.
> >> > >
> >> > > This will be bad news, I'm sorry:
> >> > >
> >> > > All remaining work on 3.x versions happens in the 3.6 branch. This
> >> > > branch is in maintenance mode.  It will only get fixes for serious
> bugs
> >> > > with no workaround.  Improvements and new features won't be
> considered
> >> > > at all.
> >> > >
> >> > > You're welcome to try backporting patches from newer issues.  Due to
> >> the
> >> > > major differences in the 3x and 4x codebases, the best case
> scenario is
> >> > > that you'll be facing a very manual task.  Some changes can't be
> >> > > backported because they rely on other features only found in 4.x
> code.
> >> > >
> >> > > Thanks,
> >> > > Shawn
> >> > >
> >> > >
> >> >
> >>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.

Greg, SOLR-4735 (using the codahale metrics lib) hasn't been committed
yet. It is still work in progress.

Actually the internal Solr Metrics class has a method to return 1
minute stats but it is not used.

On Mon, Mar 17, 2014 at 10:06 AM, Greg Pendlebury
<gr...@gmail.com> wrote:
> In the codahale metrics library there are 1, 5 and 15 minute moving
> averages just like you would see in a tool like 'top'. However in Solr I
> can only see 5 and 15 minute values, plus 'avgRequestsPerSecond'. I assumed
> this was the 1 minute value initially, but it seems to be something like
> the average since startup. I haven't looked thoroughly, but it is around 1%
> of the other two in a normally idle test cluster after load tests have been
> running for long enough that the 5 and 15 minute numbers match the load
> testing throughput.
>
> Is this difference deliberate? or an accident? or am I wrong entirely? I
> can compute the overall average anyway, given that the stats also include
> the start time of the search handler and the total search count, so I
> thought it might be an accident.
>
> Ta,
> Greg
>
>
>
>
>
> On 4 May 2013 01:19, Furkan KAMACI <fu...@gmail.com> wrote:
>
>> Does anybody tested Ganglia with JMXTrans at production environment for
>> SolrCloud?
>>
>> 2013/4/26 Dmitry Kan <so...@gmail.com>
>>
>> > Alan, Shawn,
>> >
>> > If backporting to 3.x is hard, no worries, we don't necessarily require
>> the
>> > patch as we are heading to 4.x eventually. It is just much easier within
>> > our organization to test on the existing solr 3.4 as there are a few of
>> > internal dependencies and custom code on top of solr. Also solr upgrades
>> on
>> > production systems are usually pushed forward by a month or so starting
>> the
>> > upgrade on development systems (requires lots of testing and
>> > verifications).
>> >
>> > Nevertheless, it is good effort to make #solr #graphite friendly, so keep
>> > it up! :)
>> >
>> > Dmitry
>> >
>> >
>> >
>> >
>> > On Thu, Apr 25, 2013 at 9:29 PM, Shawn Heisey <so...@elyograg.org> wrote:
>> >
>> > > On 4/25/2013 6:30 AM, Dmitry Kan wrote:
>> > > > We are very much interested in 3.4.
>> > > >
>> > > > On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk>
>> > wrote:
>> > > >> This is on top of trunk at the moment, but would be back ported to
>> 4.4
>> > > if
>> > > >> there was interest.
>> > >
>> > > This will be bad news, I'm sorry:
>> > >
>> > > All remaining work on 3.x versions happens in the 3.6 branch. This
>> > > branch is in maintenance mode.  It will only get fixes for serious bugs
>> > > with no workaround.  Improvements and new features won't be considered
>> > > at all.
>> > >
>> > > You're welcome to try backporting patches from newer issues.  Due to
>> the
>> > > major differences in the 3x and 4x codebases, the best case scenario is
>> > > that you'll be facing a very manual task.  Some changes can't be
>> > > backported because they rely on other features only found in 4.x code.
>> > >
>> > > Thanks,
>> > > Shawn
>> > >
>> > >
>> >
>>



-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Greg Pendlebury <gr...@gmail.com>.

In the codahale metrics library there are 1, 5 and 15 minute moving
averages just like you would see in a tool like 'top'. However in Solr I
can only see 5 and 15 minute values, plus 'avgRequestsPerSecond'. I assumed
this was the 1 minute value initially, but it seems to be something like
the average since startup. I haven't looked thoroughly, but it is around 1%
of the other two in a normally idle test cluster after load tests have been
running for long enough that the 5 and 15 minute numbers match the load
testing throughput.

Is this difference deliberate? or an accident? or am I wrong entirely? I
can compute the overall average anyway, given that the stats also include
the start time of the search handler and the total search count, so I
thought it might be an accident.

Ta,
Greg





On 4 May 2013 01:19, Furkan KAMACI <fu...@gmail.com> wrote:

> Does anybody tested Ganglia with JMXTrans at production environment for
> SolrCloud?
>
> 2013/4/26 Dmitry Kan <so...@gmail.com>
>
> > Alan, Shawn,
> >
> > If backporting to 3.x is hard, no worries, we don't necessarily require
> the
> > patch as we are heading to 4.x eventually. It is just much easier within
> > our organization to test on the existing solr 3.4 as there are a few of
> > internal dependencies and custom code on top of solr. Also solr upgrades
> on
> > production systems are usually pushed forward by a month or so starting
> the
> > upgrade on development systems (requires lots of testing and
> > verifications).
> >
> > Nevertheless, it is good effort to make #solr #graphite friendly, so keep
> > it up! :)
> >
> > Dmitry
> >
> >
> >
> >
> > On Thu, Apr 25, 2013 at 9:29 PM, Shawn Heisey <so...@elyograg.org> wrote:
> >
> > > On 4/25/2013 6:30 AM, Dmitry Kan wrote:
> > > > We are very much interested in 3.4.
> > > >
> > > > On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk>
> > wrote:
> > > >> This is on top of trunk at the moment, but would be back ported to
> 4.4
> > > if
> > > >> there was interest.
> > >
> > > This will be bad news, I'm sorry:
> > >
> > > All remaining work on 3.x versions happens in the 3.6 branch. This
> > > branch is in maintenance mode.  It will only get fixes for serious bugs
> > > with no workaround.  Improvements and new features won't be considered
> > > at all.
> > >
> > > You're welcome to try backporting patches from newer issues.  Due to
> the
> > > major differences in the 3x and 4x codebases, the best case scenario is
> > > that you'll be facing a very manual task.  Some changes can't be
> > > backported because they rely on other features only found in 4.x code.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Furkan KAMACI <fu...@gmail.com>.

Does anybody tested Ganglia with JMXTrans at production environment for
SolrCloud?

2013/4/26 Dmitry Kan <so...@gmail.com>

> Alan, Shawn,
>
> If backporting to 3.x is hard, no worries, we don't necessarily require the
> patch as we are heading to 4.x eventually. It is just much easier within
> our organization to test on the existing solr 3.4 as there are a few of
> internal dependencies and custom code on top of solr. Also solr upgrades on
> production systems are usually pushed forward by a month or so starting the
> upgrade on development systems (requires lots of testing and
> verifications).
>
> Nevertheless, it is good effort to make #solr #graphite friendly, so keep
> it up! :)
>
> Dmitry
>
>
>
>
> On Thu, Apr 25, 2013 at 9:29 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
> > On 4/25/2013 6:30 AM, Dmitry Kan wrote:
> > > We are very much interested in 3.4.
> > >
> > > On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk>
> wrote:
> > >> This is on top of trunk at the moment, but would be back ported to 4.4
> > if
> > >> there was interest.
> >
> > This will be bad news, I'm sorry:
> >
> > All remaining work on 3.x versions happens in the 3.6 branch. This
> > branch is in maintenance mode.  It will only get fixes for serious bugs
> > with no workaround.  Improvements and new features won't be considered
> > at all.
> >
> > You're welcome to try backporting patches from newer issues.  Due to the
> > major differences in the 3x and 4x codebases, the best case scenario is
> > that you'll be facing a very manual task.  Some changes can't be
> > backported because they rely on other features only found in 4.x code.
> >
> > Thanks,
> > Shawn
> >
> >
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Dmitry Kan <so...@gmail.com>.

Alan, Shawn,

If backporting to 3.x is hard, no worries, we don't necessarily require the
patch as we are heading to 4.x eventually. It is just much easier within
our organization to test on the existing solr 3.4 as there are a few of
internal dependencies and custom code on top of solr. Also solr upgrades on
production systems are usually pushed forward by a month or so starting the
upgrade on development systems (requires lots of testing and verifications).

Nevertheless, it is good effort to make #solr #graphite friendly, so keep
it up! :)

Dmitry

On Thu, Apr 25, 2013 at 9:29 PM, Shawn Heisey <so...@elyograg.org> wrote:

> On 4/25/2013 6:30 AM, Dmitry Kan wrote:
> > We are very much interested in 3.4.
> >
> > On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk> wrote:
> >> This is on top of trunk at the moment, but would be back ported to 4.4
> if
> >> there was interest.
>
> This will be bad news, I'm sorry:
>
> All remaining work on 3.x versions happens in the 3.6 branch. This
> branch is in maintenance mode.  It will only get fixes for serious bugs
> with no workaround.  Improvements and new features won't be considered
> at all.
>
> You're welcome to try backporting patches from newer issues.  Due to the
> major differences in the 3x and 4x codebases, the best case scenario is
> that you'll be facing a very manual task.  Some changes can't be
> backported because they rely on other features only found in 4.x code.
>
> Thanks,
> Shawn
>
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Shawn Heisey <so...@elyograg.org>.

On 4/25/2013 6:30 AM, Dmitry Kan wrote:
> We are very much interested in 3.4.
> 
> On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk> wrote:
>> This is on top of trunk at the moment, but would be back ported to 4.4 if
>> there was interest.

This will be bad news, I'm sorry:

All remaining work on 3.x versions happens in the 3.6 branch. This
branch is in maintenance mode.  It will only get fixes for serious bugs
with no workaround.  Improvements and new features won't be considered
at all.

You're welcome to try backporting patches from newer issues.  Due to the
major differences in the 3x and 4x codebases, the best case scenario is
that you'll be facing a very manual task.  Some changes can't be
backported because they rely on other features only found in 4.x code.

Thanks,
Shawn

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Dmitry Kan <so...@gmail.com>.

We are very much interested in 3.4.


On Thu, Apr 25, 2013 at 12:55 PM, Alan Woodward <al...@flax.co.uk> wrote:

> This is on top of trunk at the moment, but would be back ported to 4.4 if
> there was interest.
>
> Alan Woodward
> www.flax.co.uk
>
>
> On 25 Apr 2013, at 10:32, Dmitry Kan wrote:
>
> > Hi Alan,
> >
> > Great! What is the solr version you are patching?
> >
> > Speaking of graphite, we have set it up recently to monitor our shard
> farm.
> > So far since the RAM usage has been most important metric we were fine
> with
> > pidstat command and a little script generating stats for carbon.
> > Having some additional stats from SOLR itself would certainly be great to
> > have.
> >
> > Dmitry
> >
> > On Thu, Apr 25, 2013 at 12:01 PM, Alan Woodward <al...@flax.co.uk> wrote:
> >
> >> Hi Walter, Dmitry,
> >>
> >> I opened https://issues.apache.org/jira/browse/SOLR-4735 for this, with
> >> some work-in-progress.  Have a look!
> >>
> >> Alan Woodward
> >> www.flax.co.uk
> >>
> >>
> >> On 23 Apr 2013, at 07:40, Dmitry Kan wrote:
> >>
> >>> Hello Walter,
> >>>
> >>> Have you had a chance to get something working with graphite, codahale
> >> and
> >>> solr?
> >>>
> >>> Has anyone else tried these tools with Solr 3.x family? How much work
> is
> >> it
> >>> to set things up?
> >>>
> >>> We have tried zabbix in the past. Even though it required lots of up
> >> front
> >>> investment on configuration, it looks like a compelling option.
> >>> In the meantime, we are looking into something more "solr-tailed" yet
> >>> simple. Even without metrics persistence. Tried: jconsole and viewing
> >> stats
> >>> via jmx. Main point for us now is to gather the RAM usage.
> >>>
> >>> Dmitry
> >>>
> >>>
> >>> On Tue, Apr 9, 2013 at 9:43 PM, Walter Underwood <
> wunder@wunderwood.org
> >>> wrote:
> >>>
> >>>> If it isn't obvious, I'm glad to help test a patch for this. We can
> run
> >> a
> >>>> simulated production load in dev and report to our metrics server.
> >>>>
> >>>> wunder
> >>>>
> >>>> On Apr 8, 2013, at 1:07 PM, Walter Underwood wrote:
> >>>>
> >>>>> That approach sounds great. --wunder
> >>>>>
> >>>>> On Apr 7, 2013, at 9:40 AM, Alan Woodward wrote:
> >>>>>
> >>>>>> I've been thinking about how to improve this reporting, especially
> now
> >>>> that metrics-3 (which removes all of the funky thread issues we ran
> into
> >>>> last time I tried to add it to Solr) is close to release.  I think we
> >> could
> >>>> go about it as follows:
> >>>>>>
> >>>>>> * refactor the existing JMX reporting to use metrics-3.  This would
> >>>> mean replacing the SolrCore.infoRegistry map with a MetricsRegistry,
> and
> >>>> adding a JmxReporter, keeping the existing config logic to determine
> >> which
> >>>> JMX server to use.  PluginInfoHandler and SolrMBeanInfoHandler
> translate
> >>>> the metrics-3 data back into SolrMBean format to keep the reporting
> >>>> backwards-compatible.  This seems like a lot of work for no visible
> >>>> benefit, but…
> >>>>>> * we can then add the ability to define other metrics reporters in
> >>>> solrconfig.xml.  There are already reporters for Ganglia and Graphite
> -
> >> you
> >>>> just add then to the Solr lib/ directory, configure them in
> solrconfig,
> >> and
> >>>> voila - Solr can be monitored using the same devops tools you use to
> >>>> monitor everything else.
> >>>>>>
> >>>>>> Does this sound sane?
> >>>>>>
> >>>>>> Alan Woodward
> >>>>>> www.flax.co.uk
> >>>>>>
> >>>>>>
> >>>>>> On 6 Apr 2013, at 20:49, Walter Underwood wrote:
> >>>>>>
> >>>>>>> Wow, that really doesn't help at all, since these seem to only be
> >>>> reported in the stats page.
> >>>>>>>
> >>>>>>> I don't need another non-standard app-specific set of metrics,
> >>>> especially one that needs polling. I need metrics delivered to the
> >> common
> >>>> system that we use for all our servers.
> >>>>>>>
> >>>>>>> This is also why SPM is not useful for us, sorry Otis.
> >>>>>>>
> >>>>>>> Also, there is no time period on these stats. How do you graph the
> >>>> 95th percentile? I know there was a lot of work on these, but they
> seem
> >>>> really useless to me. I'm picky about metrics, working at Netflix does
> >> that
> >>>> to you.
> >>>>>>>
> >>>>>>> wunder
> >>>>>>>
> >>>>>>> On Apr 3, 2013, at 4:01 PM, Walter Underwood wrote:
> >>>>>>>
> >>>>>>>> In the Jira, but not in the docs.
> >>>>>>>>
> >>>>>>>> It would be nice to have VM stats like GC, too, so we can have
> >> common
> >>>> monitoring and alerting on all our services.
> >>>>>>>>
> >>>>>>>> wunder
> >>>>>>>>
> >>>>>>>> On Apr 3, 2013, at 3:31 PM, Otis Gospodnetic wrote:
> >>>>>>>>
> >>>>>>>>> It's there! :)
> >>>>>>>>>
> >> http://search-lucene.com/?q=percentile&fc_project=Solr&fc_type=issue
> >>>>>>>>>
> >>>>>>>>> Otis
> >>>>>>>>> --
> >>>>>>>>> Solr & ElasticSearch Support
> >>>>>>>>> http://sematext.com/
> >>>>>>>>>
> >>>>>>>>> On Wed, Apr 3, 2013 at 6:29 PM, Walter Underwood <
> >>>> wunder@wunderwood.org> wrote:
> >>>>>>>>>> That sounds great. I'll check out the bug, I didn't see anything
> >> in
> >>>> the docs about this. And if I can't find it with a search engine, it
> >>>> probably isn't there.  --wunder
> >>>>>>>>>>
> >>>>>>>>>> On Apr 3, 2013, at 6:39 AM, Shawn Heisey wrote:
> >>>>>>>>>>
> >>>>>>>>>>> On 3/29/2013 12:07 PM, Walter Underwood wrote:
> >>>>>>>>>>>> What are folks using for this?
> >>>>>>>>>>>
> >>>>>>>>>>> I don't know that this really answers your question, but Solr
> 4.1
> >>>> and
> >>>>>>>>>>> later includes a big chunk of codahale metrics internally for
> >>>> request
> >>>>>>>>>>> handler statistics - see SOLR-1972.  First we tried including
> the
> >>>> jar
> >>>>>>>>>>> and using the API, but that created thread leak problems, so
> the
> >>>> source
> >>>>>>>>>>> code was added.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Shawn
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> Walter Underwood
> >>>>> wunder@wunderwood.org
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Walter Underwood
> >>>> wunder@wunderwood.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Alan Woodward <al...@flax.co.uk>.

This is on top of trunk at the moment, but would be back ported to 4.4 if there was interest.

Alan Woodward
www.flax.co.uk


On 25 Apr 2013, at 10:32, Dmitry Kan wrote:

> Hi Alan,
> 
> Great! What is the solr version you are patching?
> 
> Speaking of graphite, we have set it up recently to monitor our shard farm.
> So far since the RAM usage has been most important metric we were fine with
> pidstat command and a little script generating stats for carbon.
> Having some additional stats from SOLR itself would certainly be great to
> have.
> 
> Dmitry
> 
> On Thu, Apr 25, 2013 at 12:01 PM, Alan Woodward <al...@flax.co.uk> wrote:
> 
>> Hi Walter, Dmitry,
>> 
>> I opened https://issues.apache.org/jira/browse/SOLR-4735 for this, with
>> some work-in-progress.  Have a look!
>> 
>> Alan Woodward
>> www.flax.co.uk
>> 
>> 
>> On 23 Apr 2013, at 07:40, Dmitry Kan wrote:
>> 
>>> Hello Walter,
>>> 
>>> Have you had a chance to get something working with graphite, codahale
>> and
>>> solr?
>>> 
>>> Has anyone else tried these tools with Solr 3.x family? How much work is
>> it
>>> to set things up?
>>> 
>>> We have tried zabbix in the past. Even though it required lots of up
>> front
>>> investment on configuration, it looks like a compelling option.
>>> In the meantime, we are looking into something more "solr-tailed" yet
>>> simple. Even without metrics persistence. Tried: jconsole and viewing
>> stats
>>> via jmx. Main point for us now is to gather the RAM usage.
>>> 
>>> Dmitry
>>> 
>>> 
>>> On Tue, Apr 9, 2013 at 9:43 PM, Walter Underwood <wunder@wunderwood.org
>>> wrote:
>>> 
>>>> If it isn't obvious, I'm glad to help test a patch for this. We can run
>> a
>>>> simulated production load in dev and report to our metrics server.
>>>> 
>>>> wunder
>>>> 
>>>> On Apr 8, 2013, at 1:07 PM, Walter Underwood wrote:
>>>> 
>>>>> That approach sounds great. --wunder
>>>>> 
>>>>> On Apr 7, 2013, at 9:40 AM, Alan Woodward wrote:
>>>>> 
>>>>>> I've been thinking about how to improve this reporting, especially now
>>>> that metrics-3 (which removes all of the funky thread issues we ran into
>>>> last time I tried to add it to Solr) is close to release.  I think we
>> could
>>>> go about it as follows:
>>>>>> 
>>>>>> * refactor the existing JMX reporting to use metrics-3.  This would
>>>> mean replacing the SolrCore.infoRegistry map with a MetricsRegistry, and
>>>> adding a JmxReporter, keeping the existing config logic to determine
>> which
>>>> JMX server to use.  PluginInfoHandler and SolrMBeanInfoHandler translate
>>>> the metrics-3 data back into SolrMBean format to keep the reporting
>>>> backwards-compatible.  This seems like a lot of work for no visible
>>>> benefit, but…
>>>>>> * we can then add the ability to define other metrics reporters in
>>>> solrconfig.xml.  There are already reporters for Ganglia and Graphite -
>> you
>>>> just add then to the Solr lib/ directory, configure them in solrconfig,
>> and
>>>> voila - Solr can be monitored using the same devops tools you use to
>>>> monitor everything else.
>>>>>> 
>>>>>> Does this sound sane?
>>>>>> 
>>>>>> Alan Woodward
>>>>>> www.flax.co.uk
>>>>>> 
>>>>>> 
>>>>>> On 6 Apr 2013, at 20:49, Walter Underwood wrote:
>>>>>> 
>>>>>>> Wow, that really doesn't help at all, since these seem to only be
>>>> reported in the stats page.
>>>>>>> 
>>>>>>> I don't need another non-standard app-specific set of metrics,
>>>> especially one that needs polling. I need metrics delivered to the
>> common
>>>> system that we use for all our servers.
>>>>>>> 
>>>>>>> This is also why SPM is not useful for us, sorry Otis.
>>>>>>> 
>>>>>>> Also, there is no time period on these stats. How do you graph the
>>>> 95th percentile? I know there was a lot of work on these, but they seem
>>>> really useless to me. I'm picky about metrics, working at Netflix does
>> that
>>>> to you.
>>>>>>> 
>>>>>>> wunder
>>>>>>> 
>>>>>>> On Apr 3, 2013, at 4:01 PM, Walter Underwood wrote:
>>>>>>> 
>>>>>>>> In the Jira, but not in the docs.
>>>>>>>> 
>>>>>>>> It would be nice to have VM stats like GC, too, so we can have
>> common
>>>> monitoring and alerting on all our services.
>>>>>>>> 
>>>>>>>> wunder
>>>>>>>> 
>>>>>>>> On Apr 3, 2013, at 3:31 PM, Otis Gospodnetic wrote:
>>>>>>>> 
>>>>>>>>> It's there! :)
>>>>>>>>> 
>> http://search-lucene.com/?q=percentile&fc_project=Solr&fc_type=issue
>>>>>>>>> 
>>>>>>>>> Otis
>>>>>>>>> --
>>>>>>>>> Solr & ElasticSearch Support
>>>>>>>>> http://sematext.com/
>>>>>>>>> 
>>>>>>>>> On Wed, Apr 3, 2013 at 6:29 PM, Walter Underwood <
>>>> wunder@wunderwood.org> wrote:
>>>>>>>>>> That sounds great. I'll check out the bug, I didn't see anything
>> in
>>>> the docs about this. And if I can't find it with a search engine, it
>>>> probably isn't there.  --wunder
>>>>>>>>>> 
>>>>>>>>>> On Apr 3, 2013, at 6:39 AM, Shawn Heisey wrote:
>>>>>>>>>> 
>>>>>>>>>>> On 3/29/2013 12:07 PM, Walter Underwood wrote:
>>>>>>>>>>>> What are folks using for this?
>>>>>>>>>>> 
>>>>>>>>>>> I don't know that this really answers your question, but Solr 4.1
>>>> and
>>>>>>>>>>> later includes a big chunk of codahale metrics internally for
>>>> request
>>>>>>>>>>> handler statistics - see SOLR-1972.  First we tried including the
>>>> jar
>>>>>>>>>>> and using the API, but that created thread leak problems, so the
>>>> source
>>>>>>>>>>> code was added.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Shawn
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> --
>>>>> Walter Underwood
>>>>> wunder@wunderwood.org
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Dmitry Kan <so...@gmail.com>.

Hi Alan,

Great! What is the solr version you are patching?

Speaking of graphite, we have set it up recently to monitor our shard farm.
So far since the RAM usage has been most important metric we were fine with
pidstat command and a little script generating stats for carbon.
Having some additional stats from SOLR itself would certainly be great to
have.

Dmitry

On Thu, Apr 25, 2013 at 12:01 PM, Alan Woodward <al...@flax.co.uk> wrote:

> Hi Walter, Dmitry,
>
> I opened https://issues.apache.org/jira/browse/SOLR-4735 for this, with
> some work-in-progress.  Have a look!
>
> Alan Woodward
> www.flax.co.uk
>
>
> On 23 Apr 2013, at 07:40, Dmitry Kan wrote:
>
> > Hello Walter,
> >
> > Have you had a chance to get something working with graphite, codahale
> and
> > solr?
> >
> > Has anyone else tried these tools with Solr 3.x family? How much work is
> it
> > to set things up?
> >
> > We have tried zabbix in the past. Even though it required lots of up
> front
> > investment on configuration, it looks like a compelling option.
> > In the meantime, we are looking into something more "solr-tailed" yet
> > simple. Even without metrics persistence. Tried: jconsole and viewing
> stats
> > via jmx. Main point for us now is to gather the RAM usage.
> >
> > Dmitry
> >
> >
> > On Tue, Apr 9, 2013 at 9:43 PM, Walter Underwood <wunder@wunderwood.org
> >wrote:
> >
> >> If it isn't obvious, I'm glad to help test a patch for this. We can run
> a
> >> simulated production load in dev and report to our metrics server.
> >>
> >> wunder
> >>
> >> On Apr 8, 2013, at 1:07 PM, Walter Underwood wrote:
> >>
> >>> That approach sounds great. --wunder
> >>>
> >>> On Apr 7, 2013, at 9:40 AM, Alan Woodward wrote:
> >>>
> >>>> I've been thinking about how to improve this reporting, especially now
> >> that metrics-3 (which removes all of the funky thread issues we ran into
> >> last time I tried to add it to Solr) is close to release.  I think we
> could
> >> go about it as follows:
> >>>>
> >>>> * refactor the existing JMX reporting to use metrics-3.  This would
> >> mean replacing the SolrCore.infoRegistry map with a MetricsRegistry, and
> >> adding a JmxReporter, keeping the existing config logic to determine
> which
> >> JMX server to use.  PluginInfoHandler and SolrMBeanInfoHandler translate
> >> the metrics-3 data back into SolrMBean format to keep the reporting
> >> backwards-compatible.  This seems like a lot of work for no visible
> >> benefit, but…
> >>>> * we can then add the ability to define other metrics reporters in
> >> solrconfig.xml.  There are already reporters for Ganglia and Graphite -
> you
> >> just add then to the Solr lib/ directory, configure them in solrconfig,
> and
> >> voila - Solr can be monitored using the same devops tools you use to
> >> monitor everything else.
> >>>>
> >>>> Does this sound sane?
> >>>>
> >>>> Alan Woodward
> >>>> www.flax.co.uk
> >>>>
> >>>>
> >>>> On 6 Apr 2013, at 20:49, Walter Underwood wrote:
> >>>>
> >>>>> Wow, that really doesn't help at all, since these seem to only be
> >> reported in the stats page.
> >>>>>
> >>>>> I don't need another non-standard app-specific set of metrics,
> >> especially one that needs polling. I need metrics delivered to the
> common
> >> system that we use for all our servers.
> >>>>>
> >>>>> This is also why SPM is not useful for us, sorry Otis.
> >>>>>
> >>>>> Also, there is no time period on these stats. How do you graph the
> >> 95th percentile? I know there was a lot of work on these, but they seem
> >> really useless to me. I'm picky about metrics, working at Netflix does
> that
> >> to you.
> >>>>>
> >>>>> wunder
> >>>>>
> >>>>> On Apr 3, 2013, at 4:01 PM, Walter Underwood wrote:
> >>>>>
> >>>>>> In the Jira, but not in the docs.
> >>>>>>
> >>>>>> It would be nice to have VM stats like GC, too, so we can have
> common
> >> monitoring and alerting on all our services.
> >>>>>>
> >>>>>> wunder
> >>>>>>
> >>>>>> On Apr 3, 2013, at 3:31 PM, Otis Gospodnetic wrote:
> >>>>>>
> >>>>>>> It's there! :)
> >>>>>>>
> http://search-lucene.com/?q=percentile&fc_project=Solr&fc_type=issue
> >>>>>>>
> >>>>>>> Otis
> >>>>>>> --
> >>>>>>> Solr & ElasticSearch Support
> >>>>>>> http://sematext.com/
> >>>>>>>
> >>>>>>> On Wed, Apr 3, 2013 at 6:29 PM, Walter Underwood <
> >> wunder@wunderwood.org> wrote:
> >>>>>>>> That sounds great. I'll check out the bug, I didn't see anything
> in
> >> the docs about this. And if I can't find it with a search engine, it
> >> probably isn't there.  --wunder
> >>>>>>>>
> >>>>>>>> On Apr 3, 2013, at 6:39 AM, Shawn Heisey wrote:
> >>>>>>>>
> >>>>>>>>> On 3/29/2013 12:07 PM, Walter Underwood wrote:
> >>>>>>>>>> What are folks using for this?
> >>>>>>>>>
> >>>>>>>>> I don't know that this really answers your question, but Solr 4.1
> >> and
> >>>>>>>>> later includes a big chunk of codahale metrics internally for
> >> request
> >>>>>>>>> handler statistics - see SOLR-1972.  First we tried including the
> >> jar
> >>>>>>>>> and using the API, but that created thread leak problems, so the
> >> source
> >>>>>>>>> code was added.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Shawn
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>> --
> >>> Walter Underwood
> >>> wunder@wunderwood.org
> >>>
> >>>
> >>>
> >>
> >> --
> >> Walter Underwood
> >> wunder@wunderwood.org
> >>
> >>
> >>
> >>
>
>

Re: Solr metrics in Codahale metrics and Graphite?

Posted by Alan Woodward <al...@flax.co.uk>.

Hi Walter, Dmitry,

I opened https://issues.apache.org/jira/browse/SOLR-4735 for this, with some work-in-progress.  Have a look!

Alan Woodward
www.flax.co.uk


On 23 Apr 2013, at 07:40, Dmitry Kan wrote:

> Hello Walter,
> 
> Have you had a chance to get something working with graphite, codahale and
> solr?
> 
> Has anyone else tried these tools with Solr 3.x family? How much work is it
> to set things up?
> 
> We have tried zabbix in the past. Even though it required lots of up front
> investment on configuration, it looks like a compelling option.
> In the meantime, we are looking into something more "solr-tailed" yet
> simple. Even without metrics persistence. Tried: jconsole and viewing stats
> via jmx. Main point for us now is to gather the RAM usage.
> 
> Dmitry
> 
> 
> On Tue, Apr 9, 2013 at 9:43 PM, Walter Underwood <wu...@wunderwood.org>wrote:
> 
>> If it isn't obvious, I'm glad to help test a patch for this. We can run a
>> simulated production load in dev and report to our metrics server.
>> 
>> wunder
>> 
>> On Apr 8, 2013, at 1:07 PM, Walter Underwood wrote:
>> 
>>> That approach sounds great. --wunder
>>> 
>>> On Apr 7, 2013, at 9:40 AM, Alan Woodward wrote:
>>> 
>>>> I've been thinking about how to improve this reporting, especially now
>> that metrics-3 (which removes all of the funky thread issues we ran into
>> last time I tried to add it to Solr) is close to release.  I think we could
>> go about it as follows:
>>>> 
>>>> * refactor the existing JMX reporting to use metrics-3.  This would
>> mean replacing the SolrCore.infoRegistry map with a MetricsRegistry, and
>> adding a JmxReporter, keeping the existing config logic to determine which
>> JMX server to use.  PluginInfoHandler and SolrMBeanInfoHandler translate
>> the metrics-3 data back into SolrMBean format to keep the reporting
>> backwards-compatible.  This seems like a lot of work for no visible
>> benefit, but…
>>>> * we can then add the ability to define other metrics reporters in
>> solrconfig.xml.  There are already reporters for Ganglia and Graphite - you
>> just add then to the Solr lib/ directory, configure them in solrconfig, and
>> voila - Solr can be monitored using the same devops tools you use to
>> monitor everything else.
>>>> 
>>>> Does this sound sane?
>>>> 
>>>> Alan Woodward
>>>> www.flax.co.uk
>>>> 
>>>> 
>>>> On 6 Apr 2013, at 20:49, Walter Underwood wrote:
>>>> 
>>>>> Wow, that really doesn't help at all, since these seem to only be
>> reported in the stats page.
>>>>> 
>>>>> I don't need another non-standard app-specific set of metrics,
>> especially one that needs polling. I need metrics delivered to the common
>> system that we use for all our servers.
>>>>> 
>>>>> This is also why SPM is not useful for us, sorry Otis.
>>>>> 
>>>>> Also, there is no time period on these stats. How do you graph the
>> 95th percentile? I know there was a lot of work on these, but they seem
>> really useless to me. I'm picky about metrics, working at Netflix does that
>> to you.
>>>>> 
>>>>> wunder
>>>>> 
>>>>> On Apr 3, 2013, at 4:01 PM, Walter Underwood wrote:
>>>>> 
>>>>>> In the Jira, but not in the docs.
>>>>>> 
>>>>>> It would be nice to have VM stats like GC, too, so we can have common
>> monitoring and alerting on all our services.
>>>>>> 
>>>>>> wunder
>>>>>> 
>>>>>> On Apr 3, 2013, at 3:31 PM, Otis Gospodnetic wrote:
>>>>>> 
>>>>>>> It's there! :)
>>>>>>> http://search-lucene.com/?q=percentile&fc_project=Solr&fc_type=issue
>>>>>>> 
>>>>>>> Otis
>>>>>>> --
>>>>>>> Solr & ElasticSearch Support
>>>>>>> http://sematext.com/
>>>>>>> 
>>>>>>> On Wed, Apr 3, 2013 at 6:29 PM, Walter Underwood <
>> wunder@wunderwood.org> wrote:
>>>>>>>> That sounds great. I'll check out the bug, I didn't see anything in
>> the docs about this. And if I can't find it with a search engine, it
>> probably isn't there.  --wunder
>>>>>>>> 
>>>>>>>> On Apr 3, 2013, at 6:39 AM, Shawn Heisey wrote:
>>>>>>>> 
>>>>>>>>> On 3/29/2013 12:07 PM, Walter Underwood wrote:
>>>>>>>>>> What are folks using for this?
>>>>>>>>> 
>>>>>>>>> I don't know that this really answers your question, but Solr 4.1
>> and
>>>>>>>>> later includes a big chunk of codahale metrics internally for
>> request
>>>>>>>>> handler statistics - see SOLR-1972.  First we tried including the
>> jar
>>>>>>>>> and using the API, but that created thread leak problems, so the
>> source
>>>>>>>>> code was added.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Shawn
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> --
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> 
>>> 
>>> 
>> 
>> --
>> Walter Underwood
>> wunder@wunderwood.org
>> 
>> 
>> 
>>