You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by JohnRodey <ti...@yahoo.com> on 2011/06/03 01:12:45 UTC
Hitting the URI limit, how to get around this?
I have a master solr instance that I sent my request to, it hosts no
documents it just farms the request out to a large number of shards. All the
other solr instances that host the data contain multiple cores.
Therefore my search string looks like
"http://host:port/solr/select?...&shards=nodeA:1234/solr/core01,nodeA:1234/solr/core02,nodeA:1234/solr/core03,..."
This shard list is pretty long and has finally hit "the limit".
So my question is how to best avoid having to build such a long uri?
Is there a way to have mutiple tiers, where the master server has a list of
servers (nodeA:1234,nodeB:1234,...) and each of those nodes query the cores
that they host (nodeA hosts core01, core02, core03, ...)?
Thanks!
--
View this message in context: http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-this-tp3017837p3017837.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Hitting the URI limit, how to get around this?
Posted by JohnRodey <ti...@yahoo.com>.
Yep that was my issue.
And like Ken said on Tomcat I set maxHttpHeaderSize="65536".
--
View this message in context: http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-this-tp3017837p3020774.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Hitting the URI limit, how to get around this?
Posted by Dmitry Kan <dm...@gmail.com>.
Hi,
Why not use HTTP POST?
Dmitry
On Fri, Jun 3, 2011 at 8:27 PM, Colin Bennett <cb...@job.com> wrote:
> It sounds like you need to increase the HTTP header size.
>
> In tomcat the default is 4096 bytes, and to change it you need to add
> maxHttpHeaderSize="<value>" to the connector definition in server.xml
>
> Colin.
>
> -----Original Message-----
> From: Ken Krugler [mailto:kkrugler_lists@transpac.com]
> Sent: Friday, June 03, 2011 12:39 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Hitting the URI limit, how to get around this?
>
> It sounds like you're hitting the max URL length (8K is a common default)
> for the HTTP web server that you're using to run Solr.
>
> All of the web servers I know about let you bump this limit up via
> configuration settings.
>
> -- Ken
>
> On Jun 3, 2011, at 9:27am, JohnRodey wrote:
>
> > So here's what I'm seeing: I'm running Solr 3.1
> > I'm running a java client that executes a Httpget (I tried HttpPost) with
> a
> > large shard list. If I remove a few shards from my current list it
> returns
> > fine, when I use my full shard list I get a "HTTP/1.1 400 Bad Request".
> If
> > I execute it in firefox with a few shards removed it returns fine, with
> the
> > full shard list I get a blank screen returned immediately.
> >
> > My URI works at around 7800 characters but adding one more shard to it
> blows
> > up.
> >
> > Any ideas?
> >
> > I've tried using SolrJ rather than httpget before but ran into similar
> > issues but with even less shards.
> > See
> >
>
> http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td
> 2748556.html
> >
>
> http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td
> 2748556.html
> >
> > My shards are added dynamically, every few hours I am adding new shards
> or
> > cores into the cluster. so I cannot have a shard list in the config
> files
> > unless I can somehow update them while the system is running.
> >
> > --
> > View this message in context:
>
> http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-t
> his-tp3017837p3020185.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> custom data mining solutions
>
>
>
>
>
>
>
>
>
>
--
Regards,
Dmitry Kan
RE: Hitting the URI limit, how to get around this?
Posted by Colin Bennett <cb...@job.com>.
It sounds like you need to increase the HTTP header size.
In tomcat the default is 4096 bytes, and to change it you need to add
maxHttpHeaderSize="<value>" to the connector definition in server.xml
Colin.
-----Original Message-----
From: Ken Krugler [mailto:kkrugler_lists@transpac.com]
Sent: Friday, June 03, 2011 12:39 PM
To: solr-user@lucene.apache.org
Subject: Re: Hitting the URI limit, how to get around this?
It sounds like you're hitting the max URL length (8K is a common default)
for the HTTP web server that you're using to run Solr.
All of the web servers I know about let you bump this limit up via
configuration settings.
-- Ken
On Jun 3, 2011, at 9:27am, JohnRodey wrote:
> So here's what I'm seeing: I'm running Solr 3.1
> I'm running a java client that executes a Httpget (I tried HttpPost) with
a
> large shard list. If I remove a few shards from my current list it
returns
> fine, when I use my full shard list I get a "HTTP/1.1 400 Bad Request".
If
> I execute it in firefox with a few shards removed it returns fine, with
the
> full shard list I get a blank screen returned immediately.
>
> My URI works at around 7800 characters but adding one more shard to it
blows
> up.
>
> Any ideas?
>
> I've tried using SolrJ rather than httpget before but ran into similar
> issues but with even less shards.
> See
>
http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td
2748556.html
>
http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td
2748556.html
>
> My shards are added dynamically, every few hours I am adding new shards or
> cores into the cluster. so I cannot have a shard list in the config files
> unless I can somehow update them while the system is running.
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-t
his-tp3017837p3020185.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: Hitting the URI limit, how to get around this?
Posted by Ken Krugler <kk...@transpac.com>.
It sounds like you're hitting the max URL length (8K is a common default) for the HTTP web server that you're using to run Solr.
All of the web servers I know about let you bump this limit up via configuration settings.
-- Ken
On Jun 3, 2011, at 9:27am, JohnRodey wrote:
> So here's what I'm seeing: I'm running Solr 3.1
> I'm running a java client that executes a Httpget (I tried HttpPost) with a
> large shard list. If I remove a few shards from my current list it returns
> fine, when I use my full shard list I get a "HTTP/1.1 400 Bad Request". If
> I execute it in firefox with a few shards removed it returns fine, with the
> full shard list I get a blank screen returned immediately.
>
> My URI works at around 7800 characters but adding one more shard to it blows
> up.
>
> Any ideas?
>
> I've tried using SolrJ rather than httpget before but ran into similar
> issues but with even less shards.
> See
> http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td2748556.html
> http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td2748556.html
>
> My shards are added dynamically, every few hours I am adding new shards or
> cores into the cluster. so I cannot have a shard list in the config files
> unless I can somehow update them while the system is running.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-this-tp3017837p3020185.html
> Sent from the Solr - User mailing list archive at Nabble.com.
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom data mining solutions
Re: Hitting the URI limit, how to get around this?
Posted by JohnRodey <ti...@yahoo.com>.
So here's what I'm seeing: I'm running Solr 3.1
I'm running a java client that executes a Httpget (I tried HttpPost) with a
large shard list. If I remove a few shards from my current list it returns
fine, when I use my full shard list I get a "HTTP/1.1 400 Bad Request". If
I execute it in firefox with a few shards removed it returns fine, with the
full shard list I get a blank screen returned immediately.
My URI works at around 7800 characters but adding one more shard to it blows
up.
Any ideas?
I've tried using SolrJ rather than httpget before but ran into similar
issues but with even less shards.
See
http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td2748556.html
http://lucene.472066.n3.nabble.com/Long-list-of-shards-breaks-solrj-query-td2748556.html
My shards are added dynamically, every few hours I am adding new shards or
cores into the cluster. so I cannot have a shard list in the config files
unless I can somehow update them while the system is running.
--
View this message in context: http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-this-tp3017837p3020185.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Hitting the URI limit, how to get around this?
Posted by Jayendra Patil <ja...@gmail.com>.
just a suggestion ...
If the shards are know, you can add them as the default params in the
requesthandler so they are added always. and the URL would just have
the qt parameter.
As the limit for uri is browser dependent.
How are you querying solr ...... any client api ?? through browser ??
is it hitting the max header length ?? Can you use post instead ??
Regards,
Jayendra
On Thu, Jun 2, 2011 at 7:12 PM, JohnRodey <ti...@yahoo.com> wrote:
> I have a master solr instance that I sent my request to, it hosts no
> documents it just farms the request out to a large number of shards. All the
> other solr instances that host the data contain multiple cores.
>
> Therefore my search string looks like
> "http://host:port/solr/select?...&shards=nodeA:1234/solr/core01,nodeA:1234/solr/core02,nodeA:1234/solr/core03,..."
> This shard list is pretty long and has finally hit "the limit".
>
> So my question is how to best avoid having to build such a long uri?
>
> Is there a way to have mutiple tiers, where the master server has a list of
> servers (nodeA:1234,nodeB:1234,...) and each of those nodes query the cores
> that they host (nodeA hosts core01, core02, core03, ...)?
>
> Thanks!
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Hitting-the-URI-limit-how-to-get-around-this-tp3017837p3017837.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>