You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Phil Hoy <ph...@brightsolid.com> on 2013/01/18 11:41:28 UTC
Solr load balancer
Hi,
I would like to experiment with some custom load balancers to help with query latency in the face of long gc pauses and the odd time-consuming query that we need to be able to support. At the moment setting the socket timeout via the HttpShardHandlerFactory does help, but of course it can only be set to a length of time as long as the most time consuming query we are likely to receive.
For example perhaps a load balancer that sends multiple queries concurrently to all/some replicas and only keeps the first response might be effective. Or maybe a load balancer which takes account of the frequency of timeouts would be able to recognize zombies more effectively.
To use alternative load balancer implementations cleanly and without having to hack solr directly, I would need to be able to make the existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I can then override the default load balancer using solr's plugin mechanism.
So my question is, if I made a patch to make the load balancer more pluggable, is this something that would be acceptable and if so what do I do next?
Phil
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
Re: Solr load balancer
Posted by Lance Norskog <go...@gmail.com>.
It is possible to do this with IP Multicast. The query goes out on the
multicast and all query servers read it. The servers wait for a random
amount of time, then transmit the answer. Here's the trick: it's
multicast. All of the query servers listen to each other's responses,
and drop out when another server answers the query. The server has to
decide whether to do the query before responding; this would take some
tuning.
Having all participants snoop on their peers is a really powerful
design. I worked on a telecom system that used IP Multicast to do
shortest-path-first allocation of T1 lines. Worked really well. It's a
shame Enron never used it.
On 01/24/2013 04:17 PM, Chris Hostetter wrote:
> : For example perhaps a load balancer that sends multiple queries
> : concurrently to all/some replicas and only keeps the first response
> : might be effective. Or maybe a load balancer which takes account of the
>
> I know of other distributed query systems that use this approach, when
> query speed is more important to people then load and people who use them
> seem to think it works well.
>
> given that it synthetically multiplies the load of each end user request,
> it's probably not something we'd want to turn on by default, but a
> configurable option certainly seems like it might be handy.
>
>
> -Hoss
Re: Solr load balancer
Posted by Chris Hostetter <ho...@fucit.org>.
: For example perhaps a load balancer that sends multiple queries
: concurrently to all/some replicas and only keeps the first response
: might be effective. Or maybe a load balancer which takes account of the
I know of other distributed query systems that use this approach, when
query speed is more important to people then load and people who use them
seem to think it works well.
given that it synthetically multiplies the load of each end user request,
it's probably not something we'd want to turn on by default, but a
configurable option certainly seems like it might be handy.
-Hoss
RE: Solr load balancer
Posted by Phil Hoy <ph...@brightsolid.com>.
Hi,
I have opened a couple of jira's, one to make the HttpShardHandlerFactory and LBHttpSolrServer more easily extended: https://issues.apache.org/jira/browse/SOLR-4448 and one with an implementation of a backup requesting load balancer : https://issues.apache.org/jira/browse/SOLR-4449 .
The implementation does not attempt to cancel inflight requests if a successful response is received, in fact it returns the successful response immediately then allows the inflight requests to complete. That way it can detect 'zombie' servers in a way similar to the current load balancer and not send them requests for a specified time.
Phil
-----Original Message-----
From: Jeff Wartes [mailto:jwartes@whitepages.com]
Sent: 01 February 2013 01:51
To: solr-user@lucene.apache.org
Subject: RE: Solr load balancer
For what it's worth, Google has done some pretty interesting research into coping with the idea that particular shards might very well be busy doing something else when your query comes in.
Check out this slide deck: http://research.google.com/people/jeff/latency.html
Lots of interesting ideas, but in particular, around slide 39 he talks about "backup requests" where you wait for something like your typical response time and then issue a second request to a different shard. You take whichever answer you get first, and cancel the other. The initial wait + cancellation means your extra cluster load is minimal, and you still get the benefit of reducing your p95+ response times if the first request was high-latency due to something unrelated to the query. (Say, GC.)
Of course, a central principle of this approach is being able to cancel a query and have it stop consuming resources. I'd love to be corrected, but I don't think Solr allows this. You can stop waiting for a response, but even the timeAllowed param doesn't seem to stop resource usage after the allotted time. Meaning, a few exceptionally long-running queries can take out your high-throughput cluster by tying up entire CPUs for long periods.
Let me know the JIRA number, I'd love to see work in this area.
-----Original Message-----
From: Phil Hoy [mailto:phoy@brightsolid.com]
Sent: Tuesday, January 29, 2013 11:33 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr load balancer
Hi Erick,
Thanks, I have read the blogs you cited and I found them very interesting, and we have tuned the jvm accordingly but still we get the odd longish gc pause.
That said we perhaps have an unusual setup; we index a lot of small documents using servers with ssd's and 128 GB RAM in a sharded set up with replicas and our queries rely heavily on query filters and faceting with minimal free-text style searching. For that reason we rely heavily on the filter cache to improve query latency, therefore we assign a large percentage of available ram to the jvm hosting solr.
Anyhow we are happy with the current configuration and performance profile, aside from the odd gc pause that is, and as we have index replicas it seems to me that we should be able to cope, hence my willingness to tweak how the load balancer behaves.
Thanks,
Phil
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: 20 January 2013 15:56
To: solr-user@lucene.apache.org
Subject: Re: Solr load balancer
Hmmm, the first thing I'd look at is why you are having long GC pauses. Here's a great place to start:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
and:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
I've wondered about a similar approach, but by firing off the same query to multiple nodes in your cluster, you'll be effectively doubling (at least) the load on your system. Leading to more memory issues perhaps in a "non-virtuous cycle".
FWIW,
Erick
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help with query latency in the face of long gc pauses and the odd time-consuming query that we need to be able to support. At the moment setting the socket timeout via the HttpShardHandlerFactory does help, but of course it can only be set to a length of time as long as the most time consuming query we are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries concurrently to all/some replicas and only keeps the first response might be effective. Or maybe a load balancer which takes account of the frequency of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without having to hack solr directly, I would need to be able to make the existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more pluggable, is this something that would be acceptable and if so what do I do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs ______________________________________________________________________
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
RE: Solr load balancer
Posted by Jeff Wartes <jw...@whitepages.com>.
For what it's worth, Google has done some pretty interesting research into coping with the idea that particular shards might very well be busy doing something else when your query comes in.
Check out this slide deck: http://research.google.com/people/jeff/latency.html
Lots of interesting ideas, but in particular, around slide 39 he talks about "backup requests" where you wait for something like your typical response time and then issue a second request to a different shard. You take whichever answer you get first, and cancel the other. The initial wait + cancellation means your extra cluster load is minimal, and you still get the benefit of reducing your p95+ response times if the first request was high-latency due to something unrelated to the query. (Say, GC.)
Of course, a central principle of this approach is being able to cancel a query and have it stop consuming resources. I'd love to be corrected, but I don't think Solr allows this. You can stop waiting for a response, but even the timeAllowed param doesn't seem to stop resource usage after the allotted time. Meaning, a few exceptionally long-running queries can take out your high-throughput cluster by tying up entire CPUs for long periods.
Let me know the JIRA number, I'd love to see work in this area.
-----Original Message-----
From: Phil Hoy [mailto:phoy@brightsolid.com]
Sent: Tuesday, January 29, 2013 11:33 AM
To: solr-user@lucene.apache.org
Subject: RE: Solr load balancer
Hi Erick,
Thanks, I have read the blogs you cited and I found them very interesting, and we have tuned the jvm accordingly but still we get the odd longish gc pause.
That said we perhaps have an unusual setup; we index a lot of small documents using servers with ssd's and 128 GB RAM in a sharded set up with replicas and our queries rely heavily on query filters and faceting with minimal free-text style searching. For that reason we rely heavily on the filter cache to improve query latency, therefore we assign a large percentage of available ram to the jvm hosting solr.
Anyhow we are happy with the current configuration and performance profile, aside from the odd gc pause that is, and as we have index replicas it seems to me that we should be able to cope, hence my willingness to tweak how the load balancer behaves.
Thanks,
Phil
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: 20 January 2013 15:56
To: solr-user@lucene.apache.org
Subject: Re: Solr load balancer
Hmmm, the first thing I'd look at is why you are having long GC pauses. Here's a great place to start:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
and:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
I've wondered about a similar approach, but by firing off the same query to multiple nodes in your cluster, you'll be effectively doubling (at least) the load on your system. Leading to more memory issues perhaps in a "non-virtuous cycle".
FWIW,
Erick
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help with query latency in the face of long gc pauses and the odd time-consuming query that we need to be able to support. At the moment setting the socket timeout via the HttpShardHandlerFactory does help, but of course it can only be set to a length of time as long as the most time consuming query we are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries concurrently to all/some replicas and only keeps the first response might be effective. Or maybe a load balancer which takes account of the frequency of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without having to hack solr directly, I would need to be able to make the existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more pluggable, is this something that would be acceptable and if so what do I do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
RE: Solr load balancer
Posted by Phil Hoy <ph...@brightsolid.com>.
Hi Erick,
Thanks, I have read the blogs you cited and I found them very interesting, and we have tuned the jvm accordingly but still we get the odd longish gc pause.
That said we perhaps have an unusual setup; we index a lot of small documents using servers with ssd's and 128 GB RAM in a sharded set up with replicas and our queries rely heavily on query filters and faceting with minimal free-text style searching. For that reason we rely heavily on the filter cache to improve query latency, therefore we assign a large percentage of available ram to the jvm hosting solr.
Anyhow we are happy with the current configuration and performance profile, aside from the odd gc pause that is, and as we have index replicas it seems to me that we should be able to cope, hence my willingness to tweak how the load balancer behaves.
Thanks,
Phil
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: 20 January 2013 15:56
To: solr-user@lucene.apache.org
Subject: Re: Solr load balancer
Hmmm, the first thing I'd look at is why you are having long GC pauses. Here's a great place to start:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
and:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
I've wondered about a similar approach, but by firing off the same query to multiple nodes in your cluster, you'll be effectively doubling (at least) the load on your system. Leading to more memory issues perhaps in a "non-virtuous cycle".
FWIW,
Erick
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help with query latency in the face of long gc pauses and the odd time-consuming query that we need to be able to support. At the moment setting the socket timeout via the HttpShardHandlerFactory does help, but of course it can only be set to a length of time as long as the most time consuming query we are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries concurrently to all/some replicas and only keeps the first response might be effective. Or maybe a load balancer which takes account of the frequency of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without having to hack solr directly, I would need to be able to make the existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more pluggable, is this something that would be acceptable and if so what do I do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs ______________________________________________________________________
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
Re: Solr load balancer
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, the first thing I'd look at is why you are having long GC
pauses. Here's a great place to start:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/
and:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
I've wondered about a similar approach, but by firing off the same
query to multiple nodes in your cluster, you'll be effectively
doubling (at least) the load on your system. Leading to more memory
issues perhaps in a "non-virtuous cycle".
FWIW,
Erick
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help with query latency in the face of long gc pauses and the odd time-consuming query that we need to be able to support. At the moment setting the socket timeout via the HttpShardHandlerFactory does help, but of course it can only be set to a length of time as long as the most time consuming query we are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries concurrently to all/some replicas and only keeps the first response might be effective. Or maybe a load balancer which takes account of the frequency of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without having to hack solr directly, I would need to be able to make the existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more pluggable, is this something that would be acceptable and if so what do I do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
> ______________________________________________________________________
Re: Solr load balancer
Posted by Erick Erickson <er...@gmail.com>.
About patches.
1> sure, you can open a JIRA yourself, add patches
to it, and comment etc. You have to create a signon...
2> right, once a JIRA is opened much of the discussion
moves to the JIRA, that way there's a record of what
went into the code. Any side-channel discussions should
have their conclusions recorded in the JIRA if it affects
the code.
3> What you cannot do is apply the patch to the existing
code base. Once you're happy with the patch, prod the
list (gently). Lots of committers are really swamped,
so this process can require some patience....
Best
Erick
On Thu, Jan 31, 2013 at 7:01 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> So am I correct in thinking that I add the jira myself, if so can I add it
> do the 4.2 release? Also I have further questions about the scope of my
> patch, should that be left to the comments of the jira itself?
>
> Phil
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis.gospodnetic@gmail.com]
> Sent: 22 January 2013 17:25
> To: solr-user@lucene.apache.org
> Subject: Re: Solr load balancer
>
> Hi Phil,
>
> Have a look at http://wiki.apache.org/solr/HowToContribute and thank you
> in advance! :)
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
>
> > Hi,
> >
> > I would like to experiment with some custom load balancers to help
> > with query latency in the face of long gc pauses and the odd
> > time-consuming query that we need to be able to support. At the moment
> > setting the socket timeout via the HttpShardHandlerFactory does help,
> > but of course it can only be set to a length of time as long as the
> > most time consuming query we are likely to receive.
> >
> > For example perhaps a load balancer that sends multiple queries
> > concurrently to all/some replicas and only keeps the first response
> > might be effective. Or maybe a load balancer which takes account of
> > the frequency of timeouts would be able to recognize zombies more
> effectively.
> >
> > To use alternative load balancer implementations cleanly and without
> > having to hack solr directly, I would need to be able to make the
> > existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to
> > extension, I can then override the default load balancer using solr's
> plugin mechanism.
> >
> > So my question is, if I made a patch to make the load balancer more
> > pluggable, is this something that would be acceptable and if so what
> > do I do next?
> >
> > Phil
> >
> > ______________________________________________________________________
> > "brightsolid" is used in this email to collectively mean brightsolid
> > online innovation limited and its subsidiary companies brightsolid
> > online publishing limited and brightsolid online technology limited.
> > findmypast.co.uk is a brand of brightsolid online publishing limited.
> > brightsolid online innovation limited, Gateway House, Luna Place,
> > Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No.
> SC274983.
> > brightsolid online publishing limited, The Glebe, 6 Chapel Place,
> > Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> > brightsolid online technology limited, Gateway House, Luna Place,
> > Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No.
> SC161678.
> >
> > Email Disclaimer
> >
> > This message is confidential and may contain privileged information.
> > You should not disclose its contents to any other person. If you are
> > not the intended recipient, please notify the sender named above
> > immediately. It is expressly declared that this e-mail does not
> > constitute nor form part of a contract or unilateral obligation.
> > Opinions, conclusions and other information in this message that do
> > not relate to the official business of brightsolid shall be understood
> as neither given nor endorsed by it.
> > ______________________________________________________________________
> > This email has been scanned by the brightsolid Email Security System.
> > Powered by MessageLabs
> > ______________________________________________________________________
>
>
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid
> online innovation limited and its subsidiary companies brightsolid online
> publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee
> Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place,
> Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee
> Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
>
RE: Solr load balancer
Posted by Phil Hoy <ph...@brightsolid.com>.
Hi,
So am I correct in thinking that I add the jira myself, if so can I add it do the 4.2 release? Also I have further questions about the scope of my patch, should that be left to the comments of the jira itself?
Phil
-----Original Message-----
From: Otis Gospodnetic [mailto:otis.gospodnetic@gmail.com]
Sent: 22 January 2013 17:25
To: solr-user@lucene.apache.org
Subject: Re: Solr load balancer
Hi Phil,
Have a look at http://wiki.apache.org/solr/HowToContribute and thank you in advance! :)
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help
> with query latency in the face of long gc pauses and the odd
> time-consuming query that we need to be able to support. At the moment
> setting the socket timeout via the HttpShardHandlerFactory does help,
> but of course it can only be set to a length of time as long as the
> most time consuming query we are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries
> concurrently to all/some replicas and only keeps the first response
> might be effective. Or maybe a load balancer which takes account of
> the frequency of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without
> having to hack solr directly, I would need to be able to make the
> existing LBHttpSolrServer and HttpShardHandlerFactory more amenable to
> extension, I can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more
> pluggable, is this something that would be acceptable and if so what
> do I do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid
> online innovation limited and its subsidiary companies brightsolid
> online publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place,
> Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place,
> Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place,
> Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information.
> You should not disclose its contents to any other person. If you are
> not the intended recipient, please notify the sender named above
> immediately. It is expressly declared that this e-mail does not
> constitute nor form part of a contract or unilateral obligation.
> Opinions, conclusions and other information in this message that do
> not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs ______________________________________________________________________
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
Re: Solr load balancer
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Phil,
Have a look at http://wiki.apache.org/solr/HowToContribute and thank you in
advance! :)
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Fri, Jan 18, 2013 at 5:41 AM, Phil Hoy <ph...@brightsolid.com> wrote:
> Hi,
>
> I would like to experiment with some custom load balancers to help with
> query latency in the face of long gc pauses and the odd time-consuming
> query that we need to be able to support. At the moment setting the socket
> timeout via the HttpShardHandlerFactory does help, but of course it can
> only be set to a length of time as long as the most time consuming query we
> are likely to receive.
>
> For example perhaps a load balancer that sends multiple queries
> concurrently to all/some replicas and only keeps the first response might
> be effective. Or maybe a load balancer which takes account of the frequency
> of timeouts would be able to recognize zombies more effectively.
>
> To use alternative load balancer implementations cleanly and without
> having to hack solr directly, I would need to be able to make the existing
> LBHttpSolrServer and HttpShardHandlerFactory more amenable to extension, I
> can then override the default load balancer using solr's plugin mechanism.
>
> So my question is, if I made a patch to make the load balancer more
> pluggable, is this something that would be acceptable and if so what do I
> do next?
>
> Phil
>
> ______________________________________________________________________
> "brightsolid" is used in this email to collectively mean brightsolid
> online innovation limited and its subsidiary companies brightsolid online
> publishing limited and brightsolid online technology limited.
> findmypast.co.uk is a brand of brightsolid online publishing limited.
> brightsolid online innovation limited, Gateway House, Luna Place, Dundee
> Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
> brightsolid online publishing limited, The Glebe, 6 Chapel Place,
> Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
> brightsolid online technology limited, Gateway House, Luna Place, Dundee
> Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
>
> Email Disclaimer
>
> This message is confidential and may contain privileged information. You
> should not disclose its contents to any other person. If you are not the
> intended recipient, please notify the sender named above immediately. It is
> expressly declared that this e-mail does not constitute nor form part of a
> contract or unilateral obligation. Opinions, conclusions and other
> information in this message that do not relate to the official business of
> brightsolid shall be understood as neither given nor endorsed by it.
> ______________________________________________________________________
> This email has been scanned by the brightsolid Email Security System.
> Powered by MessageLabs
> ______________________________________________________________________