You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Oliver Schrenk <os...@elmar.nl> on 2015/09/28 14:30:12 UTC
Cost of having multiple search handlers?
Hi,
I want to register multiple but identical search handler to have multiple buckets to measure performance for our different apis and consumers (and to find out who is actually using Solr).
What are there some costs associated with having multiple search handlers? Are they neglible?
Cheers,
Oliver
Re: Cost of having multiple search handlers?
Posted by Upayavira <uv...@odoko.co.uk>.
I would expect this to be negligible.
Upayavira
On Mon, Sep 28, 2015, at 01:30 PM, Oliver Schrenk wrote:
> Hi,
>
> I want to register multiple but identical search handler to have multiple
> buckets to measure performance for our different apis and consumers (and
> to find out who is actually using Solr).
>
> What are there some costs associated with having multiple search
> handlers? Are they neglible?
>
> Cheers,
> Oliver
Re: Cost of having multiple search handlers?
Posted by Jeff Wartes <jw...@whitepages.com>.
At the risk of going increasingly off-thread, yes, please do.
I’ve been using this:
https://dropwizard.github.io/metrics/3.1.0/manual/jetty/, which is
convenient, but doesn’t even have request-handler-level resolution.
Something I’ve started doing for issues that don’t seem likely to get
pulled in but also don’t really need changes in solr/lucene source code is
to publish a free-standing project (with tests) that builds the necessary
jar. For example, https://github.com/whitepages/SOLR-4449.
Seems like a decent middle ground where people can easily use or
contribute changes, and then if it gets popular enough, that’s a strong
signal it should be in the solr distribution itself.
On 9/28/15, 6:05 PM, "Walter Underwood" <wu...@wunderwood.org> wrote:
>We built our own because there was no movement on that. Don’t hold your
>breath.
>
>Glad to contribute it. We’ve been running it in production for a year,
>but the config is pretty manual.
>
>wunder
>Walter Underwood
>wunder@wunderwood.org
>http://observer.wunderwood.org/ (my blog)
>
>
>> On Sep 28, 2015, at 4:41 PM, Jeff Wartes <jw...@whitepages.com> wrote:
>>
>>
>> One would hope that https://issues.apache.org/jira/browse/SOLR-4735 will
>> be done by then.
>>
>>
>> On 9/28/15, 11:39 AM, "Walter Underwood" <wu...@wunderwood.org> wrote:
>>
>>> We did the same thing, but reporting performance metrics to Graphite.
>>>
>>> But we won’t be able to add servlet filters in 6.x, because it won’t
>>>be a
>>> webapp.
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>>
>>>
>>>> On Sep 28, 2015, at 11:32 AM, Gili Nachum <gi...@gmail.com>
>>>>wrote:
>>>>
>>>> A different solution to the same need: I'm measuring response times of
>>>> different collections measuring online/batch queries apart using New
>>>> Relic. I've added a servlet filter that analyses the request and makes
>>>> this
>>>> info available to new relic over a request argument.
>>>>
>>>> The built in new relic solr plug in doesn't provide much.
>>>> On Sep 28, 2015 17:16, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>>>
>>>>> On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
>>>>>> I want to register multiple but identical search handler to have
>>>>> multiple buckets to measure performance for our different apis and
>>>>> consumers (and to find out who is actually using Solr).
>>>>>>
>>>>>> What are there some costs associated with having multiple search
>>>>> handlers? Are they neglible?
>>>>>
>>>>> Unless you are creating hundreds or thousands of them, I doubt you'll
>>>>> notice any significant increase in resource usage from additional
>>>>> handlers. Each handler definition creates an additional URL endpoint
>>>>> within the servlet container, additional object creation within Solr,
>>>>> and perhaps an additional thread pool and threads to go with it, so
>>>>> it's
>>>>> not free, but I doubt that it's significant. The resources required
>>>>> for
>>>>> actually handling a request is likely to dwarf what's required for
>>>>>more
>>>>> handlers.
>>>>>
>>>>> Disclaimer: I have not delved into the code to figure out exactly
>>>>>what
>>>>> gets created with a search handler config, so I don't know exactly
>>>>>what
>>>>> happens. I'm basing this on general knowledge about how Java
>>>>>programs
>>>>> are constructed by expert developers, not specifics about Solr.
>>>>>
>>>>> There are others on the list who have a much better idea than I do,
>>>>>so
>>>>> if I'm wrong, I'm sure one of them will let me know.
>>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>>
>>>>>
>>>
>>
>
Re: Cost of having multiple search handlers?
Posted by Walter Underwood <wu...@wunderwood.org>.
We built our own because there was no movement on that. Don’t hold your breath.
Glad to contribute it. We’ve been running it in production for a year, but the config is pretty manual.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 28, 2015, at 4:41 PM, Jeff Wartes <jw...@whitepages.com> wrote:
>
>
> One would hope that https://issues.apache.org/jira/browse/SOLR-4735 will
> be done by then.
>
>
> On 9/28/15, 11:39 AM, "Walter Underwood" <wu...@wunderwood.org> wrote:
>
>> We did the same thing, but reporting performance metrics to Graphite.
>>
>> But we won’t be able to add servlet filters in 6.x, because it won’t be a
>> webapp.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Sep 28, 2015, at 11:32 AM, Gili Nachum <gi...@gmail.com> wrote:
>>>
>>> A different solution to the same need: I'm measuring response times of
>>> different collections measuring online/batch queries apart using New
>>> Relic. I've added a servlet filter that analyses the request and makes
>>> this
>>> info available to new relic over a request argument.
>>>
>>> The built in new relic solr plug in doesn't provide much.
>>> On Sep 28, 2015 17:16, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>>
>>>> On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
>>>>> I want to register multiple but identical search handler to have
>>>> multiple buckets to measure performance for our different apis and
>>>> consumers (and to find out who is actually using Solr).
>>>>>
>>>>> What are there some costs associated with having multiple search
>>>> handlers? Are they neglible?
>>>>
>>>> Unless you are creating hundreds or thousands of them, I doubt you'll
>>>> notice any significant increase in resource usage from additional
>>>> handlers. Each handler definition creates an additional URL endpoint
>>>> within the servlet container, additional object creation within Solr,
>>>> and perhaps an additional thread pool and threads to go with it, so
>>>> it's
>>>> not free, but I doubt that it's significant. The resources required
>>>> for
>>>> actually handling a request is likely to dwarf what's required for more
>>>> handlers.
>>>>
>>>> Disclaimer: I have not delved into the code to figure out exactly what
>>>> gets created with a search handler config, so I don't know exactly what
>>>> happens. I'm basing this on general knowledge about how Java programs
>>>> are constructed by expert developers, not specifics about Solr.
>>>>
>>>> There are others on the list who have a much better idea than I do, so
>>>> if I'm wrong, I'm sure one of them will let me know.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>>
>>
>
Re: Cost of having multiple search handlers?
Posted by Jeff Wartes <jw...@whitepages.com>.
One would hope that https://issues.apache.org/jira/browse/SOLR-4735 will
be done by then.
On 9/28/15, 11:39 AM, "Walter Underwood" <wu...@wunderwood.org> wrote:
>We did the same thing, but reporting performance metrics to Graphite.
>
>But we won’t be able to add servlet filters in 6.x, because it won’t be a
>webapp.
>
>wunder
>Walter Underwood
>wunder@wunderwood.org
>http://observer.wunderwood.org/ (my blog)
>
>
>> On Sep 28, 2015, at 11:32 AM, Gili Nachum <gi...@gmail.com> wrote:
>>
>> A different solution to the same need: I'm measuring response times of
>> different collections measuring online/batch queries apart using New
>> Relic. I've added a servlet filter that analyses the request and makes
>>this
>> info available to new relic over a request argument.
>>
>> The built in new relic solr plug in doesn't provide much.
>> On Sep 28, 2015 17:16, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>
>>> On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
>>>> I want to register multiple but identical search handler to have
>>> multiple buckets to measure performance for our different apis and
>>> consumers (and to find out who is actually using Solr).
>>>>
>>>> What are there some costs associated with having multiple search
>>> handlers? Are they neglible?
>>>
>>> Unless you are creating hundreds or thousands of them, I doubt you'll
>>> notice any significant increase in resource usage from additional
>>> handlers. Each handler definition creates an additional URL endpoint
>>> within the servlet container, additional object creation within Solr,
>>> and perhaps an additional thread pool and threads to go with it, so
>>>it's
>>> not free, but I doubt that it's significant. The resources required
>>>for
>>> actually handling a request is likely to dwarf what's required for more
>>> handlers.
>>>
>>> Disclaimer: I have not delved into the code to figure out exactly what
>>> gets created with a search handler config, so I don't know exactly what
>>> happens. I'm basing this on general knowledge about how Java programs
>>> are constructed by expert developers, not specifics about Solr.
>>>
>>> There are others on the list who have a much better idea than I do, so
>>> if I'm wrong, I'm sure one of them will let me know.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>
Re: Cost of having multiple search handlers?
Posted by Walter Underwood <wu...@wunderwood.org>.
We did the same thing, but reporting performance metrics to Graphite.
But we won’t be able to add servlet filters in 6.x, because it won’t be a webapp.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 28, 2015, at 11:32 AM, Gili Nachum <gi...@gmail.com> wrote:
>
> A different solution to the same need: I'm measuring response times of
> different collections measuring online/batch queries apart using New
> Relic. I've added a servlet filter that analyses the request and makes this
> info available to new relic over a request argument.
>
> The built in new relic solr plug in doesn't provide much.
> On Sep 28, 2015 17:16, "Shawn Heisey" <ap...@elyograg.org> wrote:
>
>> On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
>>> I want to register multiple but identical search handler to have
>> multiple buckets to measure performance for our different apis and
>> consumers (and to find out who is actually using Solr).
>>>
>>> What are there some costs associated with having multiple search
>> handlers? Are they neglible?
>>
>> Unless you are creating hundreds or thousands of them, I doubt you'll
>> notice any significant increase in resource usage from additional
>> handlers. Each handler definition creates an additional URL endpoint
>> within the servlet container, additional object creation within Solr,
>> and perhaps an additional thread pool and threads to go with it, so it's
>> not free, but I doubt that it's significant. The resources required for
>> actually handling a request is likely to dwarf what's required for more
>> handlers.
>>
>> Disclaimer: I have not delved into the code to figure out exactly what
>> gets created with a search handler config, so I don't know exactly what
>> happens. I'm basing this on general knowledge about how Java programs
>> are constructed by expert developers, not specifics about Solr.
>>
>> There are others on the list who have a much better idea than I do, so
>> if I'm wrong, I'm sure one of them will let me know.
>>
>> Thanks,
>> Shawn
>>
>>
Re: Cost of having multiple search handlers?
Posted by Gili Nachum <gi...@gmail.com>.
A different solution to the same need: I'm measuring response times of
different collections measuring online/batch queries apart using New
Relic. I've added a servlet filter that analyses the request and makes this
info available to new relic over a request argument.
The built in new relic solr plug in doesn't provide much.
On Sep 28, 2015 17:16, "Shawn Heisey" <ap...@elyograg.org> wrote:
> On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
> > I want to register multiple but identical search handler to have
> multiple buckets to measure performance for our different apis and
> consumers (and to find out who is actually using Solr).
> >
> > What are there some costs associated with having multiple search
> handlers? Are they neglible?
>
> Unless you are creating hundreds or thousands of them, I doubt you'll
> notice any significant increase in resource usage from additional
> handlers. Each handler definition creates an additional URL endpoint
> within the servlet container, additional object creation within Solr,
> and perhaps an additional thread pool and threads to go with it, so it's
> not free, but I doubt that it's significant. The resources required for
> actually handling a request is likely to dwarf what's required for more
> handlers.
>
> Disclaimer: I have not delved into the code to figure out exactly what
> gets created with a search handler config, so I don't know exactly what
> happens. I'm basing this on general knowledge about how Java programs
> are constructed by expert developers, not specifics about Solr.
>
> There are others on the list who have a much better idea than I do, so
> if I'm wrong, I'm sure one of them will let me know.
>
> Thanks,
> Shawn
>
>
Re: Cost of having multiple search handlers?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/28/2015 6:30 AM, Oliver Schrenk wrote:
> I want to register multiple but identical search handler to have multiple buckets to measure performance for our different apis and consumers (and to find out who is actually using Solr).
>
> What are there some costs associated with having multiple search handlers? Are they neglible?
Unless you are creating hundreds or thousands of them, I doubt you'll
notice any significant increase in resource usage from additional
handlers. Each handler definition creates an additional URL endpoint
within the servlet container, additional object creation within Solr,
and perhaps an additional thread pool and threads to go with it, so it's
not free, but I doubt that it's significant. The resources required for
actually handling a request is likely to dwarf what's required for more
handlers.
Disclaimer: I have not delved into the code to figure out exactly what
gets created with a search handler config, so I don't know exactly what
happens. I'm basing this on general knowledge about how Java programs
are constructed by expert developers, not specifics about Solr.
There are others on the list who have a much better idea than I do, so
if I'm wrong, I'm sure one of them will let me know.
Thanks,
Shawn