You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kent Mu <so...@gmail.com> on 2016/11/01 07:07:41 UTC
Timeout occured while waiting response from server at: http://***/solr/commodityReview
Hi friends!
We come across an issue when we use the solrj(4.9.1) to connect to solr
server, our deployment is one master with 10 replicas. we index data to the
master, and search data from the replicas via load balancing.
the error stack is as below:
*Timeout occured while waiting response from server at:
http://review.solrsearch3.cnsuning.com/solr/commodityReview
<http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://review.solrsearch3.cnsuning.com/solr/commodityReview
at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:562)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
It takes place not often. after analysis, we find that only when the
replicas Synchronous Data from master solr server. it seem that when the
replicas block search requests when synchronizing data from master, is that
true?
I wonder if it is because that our solr server hardware configuration
is too low? the physical memory is 8G with 4 cores. and the JVM we set is
Xms512m, Xmx7168m.
looking forward to your reply.
Thanks!
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Erick Erickson <er...@gmail.com>.
It really sounds like you're re-inventing SolrCloud, but
you know your requirements best.
Erick
On Wed, Nov 2, 2016 at 8:48 PM, Kent Mu <so...@gmail.com> wrote:
> Thanks Erick!
> Actually, similar to solrcloud, we split our data to 8 customized shards(1
> master with 4 slaves), and each with one ctrix and two apache web server to
> reduce server pressure through load balancing.
> As we are running an e-commerce site, the number of reviews of selling
> products grows very fast, we get the modulus on product code to put the
> reviews in the proper customized solr shard, so that we can relatively
> reduce the index size on each solr.
> we will first try to upgrade the physical memory, and let's see what it
> will happen. if the query performance is not ideal, we will try to deploy
> solr in physical machine, or we can use SSD instead.
>
> “Rome was not built in a day”, so we can explore it step by step.
> Ha ha...
> Best Regards!
> Kent
>
> 2016-11-03 1:10 GMT+08:00 Erick Erickson <er...@gmail.com>:
>
>> You need to move to SolrCloud when it's
>> time to shard ;).....
>>
>> More seriously, at some point simply adding more
>> memory will not be adequate. Either your JVM
>> heap will to grow to a point where you start encountering
>> GC pauses or the time to serve requests will
>> increase unacceptably. "when?" you ask? well
>> unfortunately there are no guidelines that can be
>> guaranteed, here's a long blog on the subject:
>>
>> https://lucidworks.com/blog/sizing-hardware-in-the-
>> abstract-why-we-dont-have-a-definitive-answer/
>>
>> The short form is you need to stress-test your
>> index and query patterns.
>>
>> Now, I've seen 20M docs strain a 32G Java heap. I've
>> seen 300M docs give very nice response times with
>> 12G of memory. It Depends (tm).
>>
>> Whether to put Solr on bare metal or not: There's
>> inevitably some penalty for a VM. That said there are lots
>> of places that use VMs successfully. Again, stress
>> testing is the key.
>>
>> And finally, using docValues for any field that sorts,
>> facets or groups will reduce the JVM requirements
>> significantly, albeit by using OS memory space, see
>> Uwe's excellent blog:
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> Best,
>> Erick
>>
>> On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <so...@gmail.com> wrote:
>> > Thanks, I got it, Erick!
>> >
>> > the size of our index data is more than 30GB every year now, and it is
>> > still growing up, and actually our solr now is running on a virtual
>> > machine. so I wonder if we need to deploy solr in a physical machine, or
>> I
>> > can just upgrade the physical memory of our Virtual machines?
>> >
>> > Best,
>> > Kent
>> >
>> > 2016-11-02 11:33 GMT+08:00 Erick Erickson <er...@gmail.com>:
>> >
>> >> Kent: OK, I see now. Then a minor pedantic point...
>> >>
>> >> It'll avoid confusion if you use master and slaves
>> >> rather than master and replicas when talking about
>> >> non-cloud setups.
>> >>
>> >> The equivalent in SolrCloud is leader and replicas.
>> >>
>> >> No big deal either way, just FYI.
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
>> >> > Thanks a lot for your reply, Shawn!
>> >> >
>> >> > no other applications on the server, I agree with you that we need to
>> >> > upgrade physical memory, and allocate the reasonable jvm size, so that
>> >> the
>> >> > operating system have spare memory available to cache the index.
>> >> >
>> >> > actually, we have nearly 100 million of data every year now, and it is
>> >> > still growing, and actually our solr now is running on a virtual
>> machine.
>> >> > so I wonder if we need to deploy solr in a physical machine.
>> >> >
>> >> > Best Regards!
>> >> > Kent
>> >> >
>> >> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
>> >> >
>> >> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
>> >> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
>> >> >> > connect to solr server, our deployment is one master with 10
>> replicas.
>> >> >> > we index data to the master, and search data from the replicas via
>> >> >> > load balancing. the error stack is as below: *Timeout occured while
>> >> >> > waiting response from server at:
>> >> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
>> >> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
>> >> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> >> >> > while waiting response from server at:
>> >> >>
>> >> >> This shows that you are connecting to port 80. It is relatively
>> rare to
>> >> >> run Solr on port 80, though it is possible. Do you have an
>> intermediate
>> >> >> layer, like a proxy or a load balancer? If so, you'll need to ensure
>> >> >> that there's not a problem there. If it works normally when
>> replication
>> >> >> isn't happening, that's probably not a worry.
>> >> >>
>> >> >> > It takes place not often. after analysis, we find that only when
>> the
>> >> >> > replicas Synchronous Data from master solr server. it seem that
>> when
>> >> >> > the replicas block search requests when synchronizing data from
>> >> >> > master, is that true?
>> >> >>
>> >> >> Solr should be able to continue serving requests while replication
>> >> >> happens. I have never heard of this happening before, and I never
>> ran
>> >> >> into it when I was using replication a long time ago on version
>> 1.4.x.
>> >> >> I think it is more likely that you've got a memory issue than a
>> bug. If
>> >> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need
>> to
>> >> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1
>> is
>> >> >> the latest at the moment, and release plans are underway for 6.3
>> right
>> >> now.
>> >> >>
>> >> >> > I wonder if it is because that our solr server hardware
>> configuration
>> >> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we
>> set
>> >> >> > is Xms512m, Xmx7168m.
>> >> >>
>> >> >> The following assumes that there is no other software on the server,
>> >> >> like a database, or an application server, web server, etc. If there
>> >> >> is, any issues are likely to be a result of extreme memory
>> starvation,
>> >> >> and possibly swapping. Additional physical memory is definitely
>> needed
>> >> >> if there is other software on the server beyond basic OS tools.
>> >> >>
>> >> >> If the total index data that is on your server is larger than about
>> 1.5
>> >> >> to 2GB, chances are excellent that you do not have enough free
>> memory to
>> >> >> cache that data effectively, which can lead to major performance
>> >> >> issues. You've only left about 1GB of memory in the system for that
>> >> >> purpose, and that memory must also run the entire operating system,
>> >> >> which can take a significant percentage of 1GB. With a large index,
>> I
>> >> >> would strongly recommend adding memory to this server.
>> >> >>
>> >> >> https://wiki.apache.org/solr/SolrPerformanceProblems
>> >> >>
>> >> >> As mentioned in that wiki page, for good performance Solr absolutely
>> >> >> requires that the operating system have spare memory available to
>> cache
>> >> >> the index. In general, allocating almost all your memory to the Java
>> >> >> heap is a bad idea with Solr.
>> >> >>
>> >> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
>> >> >> probably not necessary, unless you are doing *incredibly*
>> memory-hungry
>> >> >> queries, such as grouping, faceting, or sorting on many fields. If
>> you
>> >> >> can reduce the heap size, there would be more memory available for
>> >> caching.
>> >> >>
>> >> >> Indexing can sometimes cause very large merges to happen, and a full
>> >> >> index optimize would rewrite the entire index. Replication copies
>> the
>> >> >> changed index files, and if the size of the changes is significant,
>> >> >> additional memory can be required for good performance. See the
>> special
>> >> >> note on the wiki page above about optimizes.
>> >> >>
>> >> >> Thanks,
>> >> >> Shawn
>> >> >>
>> >> >>
>> >>
>>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Kent Mu <so...@gmail.com>.
Thanks Erick!
Actually, similar to solrcloud, we split our data to 8 customized shards(1
master with 4 slaves), and each with one ctrix and two apache web server to
reduce server pressure through load balancing.
As we are running an e-commerce site, the number of reviews of selling
products grows very fast, we get the modulus on product code to put the
reviews in the proper customized solr shard, so that we can relatively
reduce the index size on each solr.
we will first try to upgrade the physical memory, and let's see what it
will happen. if the query performance is not ideal, we will try to deploy
solr in physical machine, or we can use SSD instead.
“Rome was not built in a day”, so we can explore it step by step.
Ha ha...
Best Regards!
Kent
2016-11-03 1:10 GMT+08:00 Erick Erickson <er...@gmail.com>:
> You need to move to SolrCloud when it's
> time to shard ;).....
>
> More seriously, at some point simply adding more
> memory will not be adequate. Either your JVM
> heap will to grow to a point where you start encountering
> GC pauses or the time to serve requests will
> increase unacceptably. "when?" you ask? well
> unfortunately there are no guidelines that can be
> guaranteed, here's a long blog on the subject:
>
> https://lucidworks.com/blog/sizing-hardware-in-the-
> abstract-why-we-dont-have-a-definitive-answer/
>
> The short form is you need to stress-test your
> index and query patterns.
>
> Now, I've seen 20M docs strain a 32G Java heap. I've
> seen 300M docs give very nice response times with
> 12G of memory. It Depends (tm).
>
> Whether to put Solr on bare metal or not: There's
> inevitably some penalty for a VM. That said there are lots
> of places that use VMs successfully. Again, stress
> testing is the key.
>
> And finally, using docValues for any field that sorts,
> facets or groups will reduce the JVM requirements
> significantly, albeit by using OS memory space, see
> Uwe's excellent blog:
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Best,
> Erick
>
> On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <so...@gmail.com> wrote:
> > Thanks, I got it, Erick!
> >
> > the size of our index data is more than 30GB every year now, and it is
> > still growing up, and actually our solr now is running on a virtual
> > machine. so I wonder if we need to deploy solr in a physical machine, or
> I
> > can just upgrade the physical memory of our Virtual machines?
> >
> > Best,
> > Kent
> >
> > 2016-11-02 11:33 GMT+08:00 Erick Erickson <er...@gmail.com>:
> >
> >> Kent: OK, I see now. Then a minor pedantic point...
> >>
> >> It'll avoid confusion if you use master and slaves
> >> rather than master and replicas when talking about
> >> non-cloud setups.
> >>
> >> The equivalent in SolrCloud is leader and replicas.
> >>
> >> No big deal either way, just FYI.
> >>
> >> Best,
> >> Erick
> >>
> >> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
> >> > Thanks a lot for your reply, Shawn!
> >> >
> >> > no other applications on the server, I agree with you that we need to
> >> > upgrade physical memory, and allocate the reasonable jvm size, so that
> >> the
> >> > operating system have spare memory available to cache the index.
> >> >
> >> > actually, we have nearly 100 million of data every year now, and it is
> >> > still growing, and actually our solr now is running on a virtual
> machine.
> >> > so I wonder if we need to deploy solr in a physical machine.
> >> >
> >> > Best Regards!
> >> > Kent
> >> >
> >> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> >> >
> >> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
> >> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
> >> >> > connect to solr server, our deployment is one master with 10
> replicas.
> >> >> > we index data to the master, and search data from the replicas via
> >> >> > load balancing. the error stack is as below: *Timeout occured while
> >> >> > waiting response from server at:
> >> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
> >> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> >> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
> >> >> > while waiting response from server at:
> >> >>
> >> >> This shows that you are connecting to port 80. It is relatively
> rare to
> >> >> run Solr on port 80, though it is possible. Do you have an
> intermediate
> >> >> layer, like a proxy or a load balancer? If so, you'll need to ensure
> >> >> that there's not a problem there. If it works normally when
> replication
> >> >> isn't happening, that's probably not a worry.
> >> >>
> >> >> > It takes place not often. after analysis, we find that only when
> the
> >> >> > replicas Synchronous Data from master solr server. it seem that
> when
> >> >> > the replicas block search requests when synchronizing data from
> >> >> > master, is that true?
> >> >>
> >> >> Solr should be able to continue serving requests while replication
> >> >> happens. I have never heard of this happening before, and I never
> ran
> >> >> into it when I was using replication a long time ago on version
> 1.4.x.
> >> >> I think it is more likely that you've got a memory issue than a
> bug. If
> >> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need
> to
> >> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1
> is
> >> >> the latest at the moment, and release plans are underway for 6.3
> right
> >> now.
> >> >>
> >> >> > I wonder if it is because that our solr server hardware
> configuration
> >> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we
> set
> >> >> > is Xms512m, Xmx7168m.
> >> >>
> >> >> The following assumes that there is no other software on the server,
> >> >> like a database, or an application server, web server, etc. If there
> >> >> is, any issues are likely to be a result of extreme memory
> starvation,
> >> >> and possibly swapping. Additional physical memory is definitely
> needed
> >> >> if there is other software on the server beyond basic OS tools.
> >> >>
> >> >> If the total index data that is on your server is larger than about
> 1.5
> >> >> to 2GB, chances are excellent that you do not have enough free
> memory to
> >> >> cache that data effectively, which can lead to major performance
> >> >> issues. You've only left about 1GB of memory in the system for that
> >> >> purpose, and that memory must also run the entire operating system,
> >> >> which can take a significant percentage of 1GB. With a large index,
> I
> >> >> would strongly recommend adding memory to this server.
> >> >>
> >> >> https://wiki.apache.org/solr/SolrPerformanceProblems
> >> >>
> >> >> As mentioned in that wiki page, for good performance Solr absolutely
> >> >> requires that the operating system have spare memory available to
> cache
> >> >> the index. In general, allocating almost all your memory to the Java
> >> >> heap is a bad idea with Solr.
> >> >>
> >> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
> >> >> probably not necessary, unless you are doing *incredibly*
> memory-hungry
> >> >> queries, such as grouping, faceting, or sorting on many fields. If
> you
> >> >> can reduce the heap size, there would be more memory available for
> >> caching.
> >> >>
> >> >> Indexing can sometimes cause very large merges to happen, and a full
> >> >> index optimize would rewrite the entire index. Replication copies
> the
> >> >> changed index files, and if the size of the changes is significant,
> >> >> additional memory can be required for good performance. See the
> special
> >> >> note on the wiki page above about optimizes.
> >> >>
> >> >> Thanks,
> >> >> Shawn
> >> >>
> >> >>
> >>
>
Re: Timeout occured while waiting response from server at:
http://***/solr/commodityReview
Posted by Fuad Efendi <fu...@efendi.ca>.
My 2 cents (rounded):
Quote: "the size of our index data is more than 30GB every year now”
- is it the size of *data* or the size of *index*? This is super important!
You can have petabytes of data, growing terabytes a year, and your index files will grow only few gigabytes a year at most.
Note also that Lucene index files are immutable: it means that, for example, if your index files total size is 25Gb in a filesystem, then having at least 25Gb+2Gb of free RAM available (for index files + for OS) will be beneficial (as already mentioned in this thread).
However, caching of index files in a RAM won’t reduce search performance from minutes of response time to milliseconds. If you really have timeouts (and I believe you use at least 60 seconds timeout settings for SolrJ) then possible reasons could be:
1. “Shared VM” such as Amazon shared nodes, sometimes they just stop for few minutes
2. Garbage collection in Java
3. Sophisticated Solr query such as faceting and aggregations, with inadequately configured field cache and other caches
Having 100Gb index files in a filesystem cannot cause more than a few milliseconds response times for trivial queries such as “text:Solr”! (Exception: faceting)
You need to isolate (troubleshoot) your timeouts, and you mentioned it only happens during new queries to the new searcher after replication from Master to Slave. Which means Case #3: improperly configured cache parameters. You need warm-up query. New Solr searcher will become available after internal caches warmed up (prepopulated with data).
Memory estimate example: suppose you configured Solr such a way that it will use field cache for SKU field. Suppose SKU field is 64 bytes in average (UTF8 will take 2 bytes per character), and you have 100 millions of documents. Then you will need 6,400,000,000 bytes for just this instance of a field cache, more than 4Gb! This is basic formula. If you have few such fields, then you will need ton of memory, and you need few minutes to warm-up field cache. Calculate it properly: 8Gb or 24Gb? Consider sharding / SolrCloud if you need huge memory just for field cache. And you will be forced to consider it if you gave more that 2 billions documents (am I right? Lucene internal limitation, Integer.MAX_INT)
Thanks,
--
Fuad Efendi
(416) 993-2060
http://www.tokenizer.ca
Search Relevancy and Recommender Systems
On November 2, 2016 at 1:11:10 PM, Erick Erickson (erickerickson@gmail.com) wrote:
You need to move to SolrCloud when it's
time to shard ;).....
More seriously, at some point simply adding more
memory will not be adequate. Either your JVM
heap will to grow to a point where you start encountering
GC pauses or the time to serve requests will
increase unacceptably. "when?" you ask? well
unfortunately there are no guidelines that can be
guaranteed, here's a long blog on the subject:
https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
The short form is you need to stress-test your
index and query patterns.
Now, I've seen 20M docs strain a 32G Java heap. I've
seen 300M docs give very nice response times with
12G of memory. It Depends (tm).
Whether to put Solr on bare metal or not: There's
inevitably some penalty for a VM. That said there are lots
of places that use VMs successfully. Again, stress
testing is the key.
And finally, using docValues for any field that sorts,
facets or groups will reduce the JVM requirements
significantly, albeit by using OS memory space, see
Uwe's excellent blog:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <so...@gmail.com> wrote:
> Thanks, I got it, Erick!
>
> the size of our index data is more than 30GB every year now, and it is
> still growing up, and actually our solr now is running on a virtual
> machine. so I wonder if we need to deploy solr in a physical machine, or I
> can just upgrade the physical memory of our Virtual machines?
>
> Best,
> Kent
>
> 2016-11-02 11:33 GMT+08:00 Erick Erickson <er...@gmail.com>:
>
>> Kent: OK, I see now. Then a minor pedantic point...
>>
>> It'll avoid confusion if you use master and slaves
>> rather than master and replicas when talking about
>> non-cloud setups.
>>
>> The equivalent in SolrCloud is leader and replicas.
>>
>> No big deal either way, just FYI.
>>
>> Best,
>> Erick
>>
>> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
>> > Thanks a lot for your reply, Shawn!
>> >
>> > no other applications on the server, I agree with you that we need to
>> > upgrade physical memory, and allocate the reasonable jvm size, so that
>> the
>> > operating system have spare memory available to cache the index.
>> >
>> > actually, we have nearly 100 million of data every year now, and it is
>> > still growing, and actually our solr now is running on a virtual machine.
>> > so I wonder if we need to deploy solr in a physical machine.
>> >
>> > Best Regards!
>> > Kent
>> >
>> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
>> >
>> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
>> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
>> >> > connect to solr server, our deployment is one master with 10 replicas.
>> >> > we index data to the master, and search data from the replicas via
>> >> > load balancing. the error stack is as below: *Timeout occured while
>> >> > waiting response from server at:
>> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
>> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
>> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> >> > while waiting response from server at:
>> >>
>> >> This shows that you are connecting to port 80. It is relatively rare to
>> >> run Solr on port 80, though it is possible. Do you have an intermediate
>> >> layer, like a proxy or a load balancer? If so, you'll need to ensure
>> >> that there's not a problem there. If it works normally when replication
>> >> isn't happening, that's probably not a worry.
>> >>
>> >> > It takes place not often. after analysis, we find that only when the
>> >> > replicas Synchronous Data from master solr server. it seem that when
>> >> > the replicas block search requests when synchronizing data from
>> >> > master, is that true?
>> >>
>> >> Solr should be able to continue serving requests while replication
>> >> happens. I have never heard of this happening before, and I never ran
>> >> into it when I was using replication a long time ago on version 1.4.x.
>> >> I think it is more likely that you've got a memory issue than a bug. If
>> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
>> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
>> >> the latest at the moment, and release plans are underway for 6.3 right
>> now.
>> >>
>> >> > I wonder if it is because that our solr server hardware configuration
>> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we set
>> >> > is Xms512m, Xmx7168m.
>> >>
>> >> The following assumes that there is no other software on the server,
>> >> like a database, or an application server, web server, etc. If there
>> >> is, any issues are likely to be a result of extreme memory starvation,
>> >> and possibly swapping. Additional physical memory is definitely needed
>> >> if there is other software on the server beyond basic OS tools.
>> >>
>> >> If the total index data that is on your server is larger than about 1.5
>> >> to 2GB, chances are excellent that you do not have enough free memory to
>> >> cache that data effectively, which can lead to major performance
>> >> issues. You've only left about 1GB of memory in the system for that
>> >> purpose, and that memory must also run the entire operating system,
>> >> which can take a significant percentage of 1GB. With a large index, I
>> >> would strongly recommend adding memory to this server.
>> >>
>> >> https://wiki.apache.org/solr/SolrPerformanceProblems
>> >>
>> >> As mentioned in that wiki page, for good performance Solr absolutely
>> >> requires that the operating system have spare memory available to cache
>> >> the index. In general, allocating almost all your memory to the Java
>> >> heap is a bad idea with Solr.
>> >>
>> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
>> >> probably not necessary, unless you are doing *incredibly* memory-hungry
>> >> queries, such as grouping, faceting, or sorting on many fields. If you
>> >> can reduce the heap size, there would be more memory available for
>> caching.
>> >>
>> >> Indexing can sometimes cause very large merges to happen, and a full
>> >> index optimize would rewrite the entire index. Replication copies the
>> >> changed index files, and if the size of the changes is significant,
>> >> additional memory can be required for good performance. See the special
>> >> note on the wiki page above about optimizes.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>> >>
>>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Erick Erickson <er...@gmail.com>.
You need to move to SolrCloud when it's
time to shard ;).....
More seriously, at some point simply adding more
memory will not be adequate. Either your JVM
heap will to grow to a point where you start encountering
GC pauses or the time to serve requests will
increase unacceptably. "when?" you ask? well
unfortunately there are no guidelines that can be
guaranteed, here's a long blog on the subject:
https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
The short form is you need to stress-test your
index and query patterns.
Now, I've seen 20M docs strain a 32G Java heap. I've
seen 300M docs give very nice response times with
12G of memory. It Depends (tm).
Whether to put Solr on bare metal or not: There's
inevitably some penalty for a VM. That said there are lots
of places that use VMs successfully. Again, stress
testing is the key.
And finally, using docValues for any field that sorts,
facets or groups will reduce the JVM requirements
significantly, albeit by using OS memory space, see
Uwe's excellent blog:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <so...@gmail.com> wrote:
> Thanks, I got it, Erick!
>
> the size of our index data is more than 30GB every year now, and it is
> still growing up, and actually our solr now is running on a virtual
> machine. so I wonder if we need to deploy solr in a physical machine, or I
> can just upgrade the physical memory of our Virtual machines?
>
> Best,
> Kent
>
> 2016-11-02 11:33 GMT+08:00 Erick Erickson <er...@gmail.com>:
>
>> Kent: OK, I see now. Then a minor pedantic point...
>>
>> It'll avoid confusion if you use master and slaves
>> rather than master and replicas when talking about
>> non-cloud setups.
>>
>> The equivalent in SolrCloud is leader and replicas.
>>
>> No big deal either way, just FYI.
>>
>> Best,
>> Erick
>>
>> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
>> > Thanks a lot for your reply, Shawn!
>> >
>> > no other applications on the server, I agree with you that we need to
>> > upgrade physical memory, and allocate the reasonable jvm size, so that
>> the
>> > operating system have spare memory available to cache the index.
>> >
>> > actually, we have nearly 100 million of data every year now, and it is
>> > still growing, and actually our solr now is running on a virtual machine.
>> > so I wonder if we need to deploy solr in a physical machine.
>> >
>> > Best Regards!
>> > Kent
>> >
>> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
>> >
>> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
>> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
>> >> > connect to solr server, our deployment is one master with 10 replicas.
>> >> > we index data to the master, and search data from the replicas via
>> >> > load balancing. the error stack is as below: *Timeout occured while
>> >> > waiting response from server at:
>> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
>> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
>> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> >> > while waiting response from server at:
>> >>
>> >> This shows that you are connecting to port 80. It is relatively rare to
>> >> run Solr on port 80, though it is possible. Do you have an intermediate
>> >> layer, like a proxy or a load balancer? If so, you'll need to ensure
>> >> that there's not a problem there. If it works normally when replication
>> >> isn't happening, that's probably not a worry.
>> >>
>> >> > It takes place not often. after analysis, we find that only when the
>> >> > replicas Synchronous Data from master solr server. it seem that when
>> >> > the replicas block search requests when synchronizing data from
>> >> > master, is that true?
>> >>
>> >> Solr should be able to continue serving requests while replication
>> >> happens. I have never heard of this happening before, and I never ran
>> >> into it when I was using replication a long time ago on version 1.4.x.
>> >> I think it is more likely that you've got a memory issue than a bug. If
>> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
>> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
>> >> the latest at the moment, and release plans are underway for 6.3 right
>> now.
>> >>
>> >> > I wonder if it is because that our solr server hardware configuration
>> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we set
>> >> > is Xms512m, Xmx7168m.
>> >>
>> >> The following assumes that there is no other software on the server,
>> >> like a database, or an application server, web server, etc. If there
>> >> is, any issues are likely to be a result of extreme memory starvation,
>> >> and possibly swapping. Additional physical memory is definitely needed
>> >> if there is other software on the server beyond basic OS tools.
>> >>
>> >> If the total index data that is on your server is larger than about 1.5
>> >> to 2GB, chances are excellent that you do not have enough free memory to
>> >> cache that data effectively, which can lead to major performance
>> >> issues. You've only left about 1GB of memory in the system for that
>> >> purpose, and that memory must also run the entire operating system,
>> >> which can take a significant percentage of 1GB. With a large index, I
>> >> would strongly recommend adding memory to this server.
>> >>
>> >> https://wiki.apache.org/solr/SolrPerformanceProblems
>> >>
>> >> As mentioned in that wiki page, for good performance Solr absolutely
>> >> requires that the operating system have spare memory available to cache
>> >> the index. In general, allocating almost all your memory to the Java
>> >> heap is a bad idea with Solr.
>> >>
>> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
>> >> probably not necessary, unless you are doing *incredibly* memory-hungry
>> >> queries, such as grouping, faceting, or sorting on many fields. If you
>> >> can reduce the heap size, there would be more memory available for
>> caching.
>> >>
>> >> Indexing can sometimes cause very large merges to happen, and a full
>> >> index optimize would rewrite the entire index. Replication copies the
>> >> changed index files, and if the size of the changes is significant,
>> >> additional memory can be required for good performance. See the special
>> >> note on the wiki page above about optimizes.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>> >>
>>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Kent Mu <so...@gmail.com>.
Thanks, I got it, Erick!
the size of our index data is more than 30GB every year now, and it is
still growing up, and actually our solr now is running on a virtual
machine. so I wonder if we need to deploy solr in a physical machine, or I
can just upgrade the physical memory of our Virtual machines?
Best,
Kent
2016-11-02 11:33 GMT+08:00 Erick Erickson <er...@gmail.com>:
> Kent: OK, I see now. Then a minor pedantic point...
>
> It'll avoid confusion if you use master and slaves
> rather than master and replicas when talking about
> non-cloud setups.
>
> The equivalent in SolrCloud is leader and replicas.
>
> No big deal either way, just FYI.
>
> Best,
> Erick
>
> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
> > Thanks a lot for your reply, Shawn!
> >
> > no other applications on the server, I agree with you that we need to
> > upgrade physical memory, and allocate the reasonable jvm size, so that
> the
> > operating system have spare memory available to cache the index.
> >
> > actually, we have nearly 100 million of data every year now, and it is
> > still growing, and actually our solr now is running on a virtual machine.
> > so I wonder if we need to deploy solr in a physical machine.
> >
> > Best Regards!
> > Kent
> >
> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> >
> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
> >> > connect to solr server, our deployment is one master with 10 replicas.
> >> > we index data to the master, and search data from the replicas via
> >> > load balancing. the error stack is as below: *Timeout occured while
> >> > waiting response from server at:
> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
> >> > while waiting response from server at:
> >>
> >> This shows that you are connecting to port 80. It is relatively rare to
> >> run Solr on port 80, though it is possible. Do you have an intermediate
> >> layer, like a proxy or a load balancer? If so, you'll need to ensure
> >> that there's not a problem there. If it works normally when replication
> >> isn't happening, that's probably not a worry.
> >>
> >> > It takes place not often. after analysis, we find that only when the
> >> > replicas Synchronous Data from master solr server. it seem that when
> >> > the replicas block search requests when synchronizing data from
> >> > master, is that true?
> >>
> >> Solr should be able to continue serving requests while replication
> >> happens. I have never heard of this happening before, and I never ran
> >> into it when I was using replication a long time ago on version 1.4.x.
> >> I think it is more likely that you've got a memory issue than a bug. If
> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
> >> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
> >> the latest at the moment, and release plans are underway for 6.3 right
> now.
> >>
> >> > I wonder if it is because that our solr server hardware configuration
> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we set
> >> > is Xms512m, Xmx7168m.
> >>
> >> The following assumes that there is no other software on the server,
> >> like a database, or an application server, web server, etc. If there
> >> is, any issues are likely to be a result of extreme memory starvation,
> >> and possibly swapping. Additional physical memory is definitely needed
> >> if there is other software on the server beyond basic OS tools.
> >>
> >> If the total index data that is on your server is larger than about 1.5
> >> to 2GB, chances are excellent that you do not have enough free memory to
> >> cache that data effectively, which can lead to major performance
> >> issues. You've only left about 1GB of memory in the system for that
> >> purpose, and that memory must also run the entire operating system,
> >> which can take a significant percentage of 1GB. With a large index, I
> >> would strongly recommend adding memory to this server.
> >>
> >> https://wiki.apache.org/solr/SolrPerformanceProblems
> >>
> >> As mentioned in that wiki page, for good performance Solr absolutely
> >> requires that the operating system have spare memory available to cache
> >> the index. In general, allocating almost all your memory to the Java
> >> heap is a bad idea with Solr.
> >>
> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
> >> probably not necessary, unless you are doing *incredibly* memory-hungry
> >> queries, such as grouping, faceting, or sorting on many fields. If you
> >> can reduce the heap size, there would be more memory available for
> caching.
> >>
> >> Indexing can sometimes cause very large merges to happen, and a full
> >> index optimize would rewrite the entire index. Replication copies the
> >> changed index files, and if the size of the changes is significant,
> >> additional memory can be required for good performance. See the special
> >> note on the wiki page above about optimizes.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Erick Erickson <er...@gmail.com>.
Kent: OK, I see now. Then a minor pedantic point...
It'll avoid confusion if you use master and slaves
rather than master and replicas when talking about
non-cloud setups.
The equivalent in SolrCloud is leader and replicas.
No big deal either way, just FYI.
Best,
Erick
On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <so...@gmail.com> wrote:
> Thanks a lot for your reply, Shawn!
>
> no other applications on the server, I agree with you that we need to
> upgrade physical memory, and allocate the reasonable jvm size, so that the
> operating system have spare memory available to cache the index.
>
> actually, we have nearly 100 million of data every year now, and it is
> still growing, and actually our solr now is running on a virtual machine.
> so I wonder if we need to deploy solr in a physical machine.
>
> Best Regards!
> Kent
>
> 2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
>
>> On 11/1/2016 1:07 AM, Kent Mu wrote:
>> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
>> > connect to solr server, our deployment is one master with 10 replicas.
>> > we index data to the master, and search data from the replicas via
>> > load balancing. the error stack is as below: *Timeout occured while
>> > waiting response from server at:
>> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
>> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
>> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> > while waiting response from server at:
>>
>> This shows that you are connecting to port 80. It is relatively rare to
>> run Solr on port 80, though it is possible. Do you have an intermediate
>> layer, like a proxy or a load balancer? If so, you'll need to ensure
>> that there's not a problem there. If it works normally when replication
>> isn't happening, that's probably not a worry.
>>
>> > It takes place not often. after analysis, we find that only when the
>> > replicas Synchronous Data from master solr server. it seem that when
>> > the replicas block search requests when synchronizing data from
>> > master, is that true?
>>
>> Solr should be able to continue serving requests while replication
>> happens. I have never heard of this happening before, and I never ran
>> into it when I was using replication a long time ago on version 1.4.x.
>> I think it is more likely that you've got a memory issue than a bug. If
>> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
>> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
>> the latest at the moment, and release plans are underway for 6.3 right now.
>>
>> > I wonder if it is because that our solr server hardware configuration
>> > is too low? the physical memory is 8G with 4 cores. and the JVM we set
>> > is Xms512m, Xmx7168m.
>>
>> The following assumes that there is no other software on the server,
>> like a database, or an application server, web server, etc. If there
>> is, any issues are likely to be a result of extreme memory starvation,
>> and possibly swapping. Additional physical memory is definitely needed
>> if there is other software on the server beyond basic OS tools.
>>
>> If the total index data that is on your server is larger than about 1.5
>> to 2GB, chances are excellent that you do not have enough free memory to
>> cache that data effectively, which can lead to major performance
>> issues. You've only left about 1GB of memory in the system for that
>> purpose, and that memory must also run the entire operating system,
>> which can take a significant percentage of 1GB. With a large index, I
>> would strongly recommend adding memory to this server.
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems
>>
>> As mentioned in that wiki page, for good performance Solr absolutely
>> requires that the operating system have spare memory available to cache
>> the index. In general, allocating almost all your memory to the Java
>> heap is a bad idea with Solr.
>>
>> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
>> probably not necessary, unless you are doing *incredibly* memory-hungry
>> queries, such as grouping, faceting, or sorting on many fields. If you
>> can reduce the heap size, there would be more memory available for caching.
>>
>> Indexing can sometimes cause very large merges to happen, and a full
>> index optimize would rewrite the entire index. Replication copies the
>> changed index files, and if the size of the changes is significant,
>> additional memory can be required for good performance. See the special
>> note on the wiki page above about optimizes.
>>
>> Thanks,
>> Shawn
>>
>>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Kent Mu <so...@gmail.com>.
Thanks a lot for your reply, Shawn!
no other applications on the server, I agree with you that we need to
upgrade physical memory, and allocate the reasonable jvm size, so that the
operating system have spare memory available to cache the index.
actually, we have nearly 100 million of data every year now, and it is
still growing, and actually our solr now is running on a virtual machine.
so I wonder if we need to deploy solr in a physical machine.
Best Regards!
Kent
2016-11-01 21:18 GMT+08:00 Shawn Heisey <ap...@elyograg.org>:
> On 11/1/2016 1:07 AM, Kent Mu wrote:
> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
> > connect to solr server, our deployment is one master with 10 replicas.
> > we index data to the master, and search data from the replicas via
> > load balancing. the error stack is as below: *Timeout occured while
> > waiting response from server at:
> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
> > while waiting response from server at:
>
> This shows that you are connecting to port 80. It is relatively rare to
> run Solr on port 80, though it is possible. Do you have an intermediate
> layer, like a proxy or a load balancer? If so, you'll need to ensure
> that there's not a problem there. If it works normally when replication
> isn't happening, that's probably not a worry.
>
> > It takes place not often. after analysis, we find that only when the
> > replicas Synchronous Data from master solr server. it seem that when
> > the replicas block search requests when synchronizing data from
> > master, is that true?
>
> Solr should be able to continue serving requests while replication
> happens. I have never heard of this happening before, and I never ran
> into it when I was using replication a long time ago on version 1.4.x.
> I think it is more likely that you've got a memory issue than a bug. If
> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
> upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
> the latest at the moment, and release plans are underway for 6.3 right now.
>
> > I wonder if it is because that our solr server hardware configuration
> > is too low? the physical memory is 8G with 4 cores. and the JVM we set
> > is Xms512m, Xmx7168m.
>
> The following assumes that there is no other software on the server,
> like a database, or an application server, web server, etc. If there
> is, any issues are likely to be a result of extreme memory starvation,
> and possibly swapping. Additional physical memory is definitely needed
> if there is other software on the server beyond basic OS tools.
>
> If the total index data that is on your server is larger than about 1.5
> to 2GB, chances are excellent that you do not have enough free memory to
> cache that data effectively, which can lead to major performance
> issues. You've only left about 1GB of memory in the system for that
> purpose, and that memory must also run the entire operating system,
> which can take a significant percentage of 1GB. With a large index, I
> would strongly recommend adding memory to this server.
>
> https://wiki.apache.org/solr/SolrPerformanceProblems
>
> As mentioned in that wiki page, for good performance Solr absolutely
> requires that the operating system have spare memory available to cache
> the index. In general, allocating almost all your memory to the Java
> heap is a bad idea with Solr.
>
> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
> probably not necessary, unless you are doing *incredibly* memory-hungry
> queries, such as grouping, faceting, or sorting on many fields. If you
> can reduce the heap size, there would be more memory available for caching.
>
> Indexing can sometimes cause very large merges to happen, and a full
> index optimize would rewrite the entire index. Replication copies the
> changed index files, and if the size of the changes is significant,
> additional memory can be required for good performance. See the special
> note on the wiki page above about optimizes.
>
> Thanks,
> Shawn
>
>
Re: Timeout occured while waiting response from server at:
http://***/solr/commodityReview
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/1/2016 1:07 AM, Kent Mu wrote:
> Hi friends! We come across an issue when we use the solrj(4.9.1) to
> connect to solr server, our deployment is one master with 10 replicas.
> we index data to the master, and search data from the replicas via
> load balancing. the error stack is as below: *Timeout occured while
> waiting response from server at:
> http://review.solrsearch3.cnsuning.com/solr/commodityReview
> <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> org.apache.solr.client.solrj.SolrServerException: Timeout occured
> while waiting response from server at:
This shows that you are connecting to port 80. It is relatively rare to
run Solr on port 80, though it is possible. Do you have an intermediate
layer, like a proxy or a load balancer? If so, you'll need to ensure
that there's not a problem there. If it works normally when replication
isn't happening, that's probably not a worry.
> It takes place not often. after analysis, we find that only when the
> replicas Synchronous Data from master solr server. it seem that when
> the replicas block search requests when synchronizing data from
> master, is that true?
Solr should be able to continue serving requests while replication
happens. I have never heard of this happening before, and I never ran
into it when I was using replication a long time ago on version 1.4.x.
I think it is more likely that you've got a memory issue than a bug. If
it IS a bug, it will *not* be fixed in a 4.x version, you would need to
upgrade to 6.x and see whether it's still a problem. Version 6.2.1 is
the latest at the moment, and release plans are underway for 6.3 right now.
> I wonder if it is because that our solr server hardware configuration
> is too low? the physical memory is 8G with 4 cores. and the JVM we set
> is Xms512m, Xmx7168m.
The following assumes that there is no other software on the server,
like a database, or an application server, web server, etc. If there
is, any issues are likely to be a result of extreme memory starvation,
and possibly swapping. Additional physical memory is definitely needed
if there is other software on the server beyond basic OS tools.
If the total index data that is on your server is larger than about 1.5
to 2GB, chances are excellent that you do not have enough free memory to
cache that data effectively, which can lead to major performance
issues. You've only left about 1GB of memory in the system for that
purpose, and that memory must also run the entire operating system,
which can take a significant percentage of 1GB. With a large index, I
would strongly recommend adding memory to this server.
https://wiki.apache.org/solr/SolrPerformanceProblems
As mentioned in that wiki page, for good performance Solr absolutely
requires that the operating system have spare memory available to cache
the index. In general, allocating almost all your memory to the Java
heap is a bad idea with Solr.
If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
probably not necessary, unless you are doing *incredibly* memory-hungry
queries, such as grouping, faceting, or sorting on many fields. If you
can reduce the heap size, there would be more memory available for caching.
Indexing can sometimes cause very large merges to happen, and a full
index optimize would rewrite the entire index. Replication copies the
changed index files, and if the size of the changes is significant,
additional memory can be required for good performance. See the special
note on the wiki page above about optimizes.
Thanks,
Shawn
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Kent Mu <so...@gmail.com>.
well, we do not use solrcloud, just simple solr deployment - one master
with some relicas. I agree with Shawn's opinion, I think we need to upgrade
the physical memory, and allocate the reasonable jvm size.
Thank you all the same!
Best Regards!
Kent
2016-11-02 4:52 GMT+08:00 Erick Erickson <er...@gmail.com>:
> This is a bit confusing as you're mixing terms from
> older master/slave Solr with SolrCloud.
>
> You say "our deployment is one master with 10 replicas"
> and
> "we index data to the
> master, and search data from the replicas via load balancing"
>
> So how are you getting your data to the replicas?
>
> There is no "master" in SolCloud, just a "leader" whose
> role as assumed by one replica for each shard.
>
> However, if you are issuing a "fetchindex" via
> the replication API to move the data to an
> independent search cluster, your might be seeing:
>
> https://issues.apache.org/jira/browse/SOLR-9706
>
> ---------------
> Now, the only real time I've seen this be necessary
> is in _extremely_ heavy indexing situations. Otherwise
> you should just have a single solrCloud that you index
> to and search from.
>
> Or do you have some setup where each replica acts
> as a slave to some replica in your indexing cluster? I
> suspect that runs into the same situation as described
> in SOLR-9706.
>
> In the "normal" case, the only time a replica should
> do a fetchindex is when it should be in a "recovering"
> state and not be serving any queries anyway.
>
> ------------
> Again, though, unless it's a rather unusual setup you
> shouldn't need to do this.
>
> Best,
> Erick
>
> On Tue, Nov 1, 2016 at 9:46 AM, Fuad Efendi <fu...@efendi.ca> wrote:
> > Quote:
> > It takes place not often. after analysis, we find that only when the
> > replicas Synchronous Data from master solr server. it seem that when the
> > replicas block search requests when synchronizing data from master, is
> that
> > true?
> >
> >
> > Solr makes new searcher available after replication complete, and new
> *trivial* searches should take milliseconds of response time even with zero
> cache tunings including OS managed caches for filesystem.
> >
> > However, if first search coming uses faceting (which uses field caches)
> then it may takes from seconds to minutes to many minutes just to warm up
> internal caches.
> >
> > Solr has the way to warm up internal caches before making new searcher
> available: https://cwiki.apache.org/confluence/display/solr/Query+
> Settings+in+SolrConfig
> >
> > Make this queries typical for your use cases (for instance, *:* with
> faceting):
> >
> > <listener event="newSearcher" class="solr.QuerySenderListener">
> > <arr name="queries">
> > <!--
> > <lst><str name="q">solr</str><str name="sort">price asc</str></lst>
> > <lst><str name="q">rocks</str><str name="sort">weight asc</str></lst>
> > -->
> > </arr>
> > </listener>
> >
> >
> >
> > Thanks,
> >
> > --
> > Fuad Efendi
> > (416) 993-2060
> > http://www.tokenizer.ca
> > Search Relevancy and Recommender Systems
> >
> >
> > On November 1, 2016 at 12:07:50 PM, Kent Mu (solr.study@gmail.com)
> wrote:
> >
> > Hi friends!
> > We come across an issue when we use the solrj(4.9.1) to connect to solr
> > server, our deployment is one master with 10 replicas. we index data to
> the
> > master, and search data from the replicas via load balancing.
> >
> > the error stack is as below:
> >
> > *Timeout occured while waiting response from server at:
> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> > org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> > waiting response from server at:
> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrServer.
> executeMethod(HttpSolrServer.java:562)
> > ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(
> HttpSolrServer.java:210)
> > ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(
> HttpSolrServer.java:206)
> > ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> > at
> > org.apache.solr.client.solrj.request.QueryRequest.process(
> QueryRequest.java:91)
> > ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
> > ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> >
> > It takes place not often. after analysis, we find that only when the
> > replicas Synchronous Data from master solr server. it seem that when the
> > replicas block search requests when synchronizing data from master, is
> that
> > true?
> > I wonder if it is because that our solr server hardware configuration
> > is too low? the physical memory is 8G with 4 cores. and the JVM we set is
> > Xms512m, Xmx7168m.
> >
> > looking forward to your reply.
> >
> > Thanks!
>
Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Posted by Erick Erickson <er...@gmail.com>.
This is a bit confusing as you're mixing terms from
older master/slave Solr with SolrCloud.
You say "our deployment is one master with 10 replicas"
and
"we index data to the
master, and search data from the replicas via load balancing"
So how are you getting your data to the replicas?
There is no "master" in SolCloud, just a "leader" whose
role as assumed by one replica for each shard.
However, if you are issuing a "fetchindex" via
the replication API to move the data to an
independent search cluster, your might be seeing:
https://issues.apache.org/jira/browse/SOLR-9706
---------------
Now, the only real time I've seen this be necessary
is in _extremely_ heavy indexing situations. Otherwise
you should just have a single solrCloud that you index
to and search from.
Or do you have some setup where each replica acts
as a slave to some replica in your indexing cluster? I
suspect that runs into the same situation as described
in SOLR-9706.
In the "normal" case, the only time a replica should
do a fetchindex is when it should be in a "recovering"
state and not be serving any queries anyway.
------------
Again, though, unless it's a rather unusual setup you
shouldn't need to do this.
Best,
Erick
On Tue, Nov 1, 2016 at 9:46 AM, Fuad Efendi <fu...@efendi.ca> wrote:
> Quote:
> It takes place not often. after analysis, we find that only when the
> replicas Synchronous Data from master solr server. it seem that when the
> replicas block search requests when synchronizing data from master, is that
> true?
>
>
> Solr makes new searcher available after replication complete, and new *trivial* searches should take milliseconds of response time even with zero cache tunings including OS managed caches for filesystem.
>
> However, if first search coming uses faceting (which uses field caches) then it may takes from seconds to minutes to many minutes just to warm up internal caches.
>
> Solr has the way to warm up internal caches before making new searcher available: https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+SolrConfig
>
> Make this queries typical for your use cases (for instance, *:* with faceting):
>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <!--
> <lst><str name="q">solr</str><str name="sort">price asc</str></lst>
> <lst><str name="q">rocks</str><str name="sort">weight asc</str></lst>
> -->
> </arr>
> </listener>
>
>
>
> Thanks,
>
> --
> Fuad Efendi
> (416) 993-2060
> http://www.tokenizer.ca
> Search Relevancy and Recommender Systems
>
>
> On November 1, 2016 at 12:07:50 PM, Kent Mu (solr.study@gmail.com) wrote:
>
> Hi friends!
> We come across an issue when we use the solrj(4.9.1) to connect to solr
> server, our deployment is one master with 10 replicas. we index data to the
> master, and search data from the replicas via load balancing.
>
> the error stack is as below:
>
> *Timeout occured while waiting response from server at:
> http://review.solrsearch3.cnsuning.com/solr/commodityReview
> <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://review.solrsearch3.cnsuning.com/solr/commodityReview
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:562)
> ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
> ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
> ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
> ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
> at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
> ~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
>
> It takes place not often. after analysis, we find that only when the
> replicas Synchronous Data from master solr server. it seem that when the
> replicas block search requests when synchronizing data from master, is that
> true?
> I wonder if it is because that our solr server hardware configuration
> is too low? the physical memory is 8G with 4 cores. and the JVM we set is
> Xms512m, Xmx7168m.
>
> looking forward to your reply.
>
> Thanks!
Re: Timeout occured while waiting response from server at:
http://***/solr/commodityReview
Posted by Fuad Efendi <fu...@efendi.ca>.
Quote:
It takes place not often. after analysis, we find that only when the
replicas Synchronous Data from master solr server. it seem that when the
replicas block search requests when synchronizing data from master, is that
true?
Solr makes new searcher available after replication complete, and new *trivial* searches should take milliseconds of response time even with zero cache tunings including OS managed caches for filesystem.
However, if first search coming uses faceting (which uses field caches) then it may takes from seconds to minutes to many minutes just to warm up internal caches.
Solr has the way to warm up internal caches before making new searcher available: https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+SolrConfig
Make this queries typical for your use cases (for instance, *:* with faceting):
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!--
<lst><str name="q">solr</str><str name="sort">price asc</str></lst>
<lst><str name="q">rocks</str><str name="sort">weight asc</str></lst>
-->
</arr>
</listener>
Thanks,
--
Fuad Efendi
(416) 993-2060
http://www.tokenizer.ca
Search Relevancy and Recommender Systems
On November 1, 2016 at 12:07:50 PM, Kent Mu (solr.study@gmail.com) wrote:
Hi friends!
We come across an issue when we use the solrj(4.9.1) to connect to solr
server, our deployment is one master with 10 replicas. we index data to the
master, and search data from the replicas via load balancing.
the error stack is as below:
*Timeout occured while waiting response from server at:
http://review.solrsearch3.cnsuning.com/solr/commodityReview
<http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://review.solrsearch3.cnsuning.com/solr/commodityReview
at
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:562)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
~[solr-solrj-4.9.1.jar:4.9.1 1625909 - mike - 2014-09-18 04:09:05]
It takes place not often. after analysis, we find that only when the
replicas Synchronous Data from master solr server. it seem that when the
replicas block search requests when synchronizing data from master, is that
true?
I wonder if it is because that our solr server hardware configuration
is too low? the physical memory is 8G with 4 cores. and the JVM we set is
Xms512m, Xmx7168m.
looking forward to your reply.
Thanks!