You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Simonian, Marta M (US SSA)" <ma...@baesystems.com> on 2013/07/31 00:43:36 UTC
Ingesting geo data into Solr very slow
Hi,
We are using Solr 4.4 to ingest geo data and it's really slow. When we don't index the geo it takes seconds to ingest 100, 000 records but as soon as we add it takes 2 hours.
Also we found that when changing the distErrPct from 0.025 to 0.1, 1000 rows are ingested in 20 sec vs 2 min. But we can't change that setting as we want our search to be as accurate as possible.
About the environment we are running Solr on 6 CPUs and 8GB of memory. We've been monitoring the VMs and they seem to be ok.
We are running on Tomcat but we might switch to Jetty to see if that will increase the performance.
We use ConcurrentUpdateSolrServer(httpSolrServer, 5000, 2);
We are saving a bean rather than a solr document (server.addBean(item)). I'm not sure if that could make it slow as it's going to do some conversion?
Can you please let me know what are the best settings for Solr? Maybe some changes in the solrconfig.xml or the schema.xml?
What are the preferred environment settings and resources?
Thank you!
Marta
RE: Ingesting geo data into Solr very slow
Posted by "Simonian, Marta M (US SSA)" <ma...@baesystems.com>.
Hi guys,
Here is the reply I got from the solr group. I'll change those settings. It's good to know that it doesn't matter if we use the bean vs solr doc.
-Marta
-----Original Message-----
From: David Smiley (@MITRE.org) [mailto:DSMILEY@mitre.org]
Sent: Tuesday, July 30, 2013 9:08 PM
To: solr-user@lucene.apache.org
Subject: Re: Ingesting geo data into Solr very slow
Hi Marta,
Presumably you are indexing polygons -- I suspect complex ones. There isn't too much that you can do about this right now other than index them in parallel. I see you are doing this in 2 threads; try 4, or maybe even 6.
Also, ensure that maxDistErr is reflective of the smallest distance you need to distinguish between. It may help a little but not much. I can think of some internal code details that might be improved but that doesn't help you now.
There's some generic Solr things you can do to improve indexing performance too like increasing the indexing buffer size (100MB -> 200MB) and the mergeFactor (10->20 albeit temporarily and/or issue optimize), both in solrconfig.xml.
Changing the servlet engine won't help. Calling server.addBean(item) isn't a problem either.
~ David
Simonian, Marta M (US SSA) wrote
> Hi,
>
> We are using Solr 4.4 to ingest geo data and it's really slow. When we
> don't index the geo it takes seconds to ingest 100, 000 records but as
> soon as we add it takes 2 hours.
>
> Also we found that when changing the distErrPct from 0.025 to 0.1,
> 1000 rows are ingested in 20 sec vs 2 min. But we can't change that
> setting as we want our search to be as accurate as possible.
>
> About the environment we are running Solr on 6 CPUs and 8GB of memory.
> We've been monitoring the VMs and they seem to be ok.
>
> We are running on Tomcat but we might switch to Jetty to see if that
> will increase the performance.
>
> We use ConcurrentUpdateSolrServer(httpSolrServer, 5000, 2);
>
> We are saving a bean rather than a solr document (server.addBean(item)).
> I'm not sure if that could make it slow as it's going to do some
> conversion?
>
> Can you please let me know what are the best settings for Solr? Maybe
> some changes in the solrconfig.xml or the schema.xml?
> What are the preferred environment settings and resources?
>
> Thank you!
> Marta
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Ingesting-geo-data-into-Solr-very-slow-tp4081484p4081527.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Ingesting geo data into Solr very slow
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/31/2013 11:20 AM, Simonian, Marta M (US SSA) wrote:
> Does anybody know if Solr performs better on Jetty vs Tomcat?
Jetty has less complexity than tomcat. It is likely to use less memory.
If you went with default settings for both, jetty is likely to perform
better, but the difference would probably be very small.
If you understand how to tune your servlet container, then there's no
way to answer that question. You should use whatever you are
comfortable with. A well-tuned tomcat server would probably perform
better than the default example jetty - but you have to do that tuning.
The only concrete information I can give you is this: Solr tests use
jetty, so jetty is the only container that is fully tested with Solr.
Bugs *have* been found with other containers, and they get fixed as fast
as possible.
The other point worth reiterating: Unless you carefully tune your
container, something this list can't really help you with, the container
choice probably isn't going to affect performance much.
Thanks,
Shawn
RE: Ingesting geo data into Solr very slow
Posted by "Simonian, Marta M (US SSA)" <ma...@baesystems.com>.
Does anybody know if Solr performs better on Jetty vs Tomcat?
-----Original Message-----
From: David Smiley (@MITRE.org) [mailto:DSMILEY@mitre.org]
Sent: Tuesday, July 30, 2013 9:08 PM
To: solr-user@lucene.apache.org
Subject: Re: Ingesting geo data into Solr very slow
Hi Marta,
Presumably you are indexing polygons -- I suspect complex ones. There isn't too much that you can do about this right now other than index them in parallel. I see you are doing this in 2 threads; try 4, or maybe even 6.
Also, ensure that maxDistErr is reflective of the smallest distance you need to distinguish between. It may help a little but not much. I can think of some internal code details that might be improved but that doesn't help you now.
There's some generic Solr things you can do to improve indexing performance too like increasing the indexing buffer size (100MB -> 200MB) and the mergeFactor (10->20 albeit temporarily and/or issue optimize), both in solrconfig.xml.
Changing the servlet engine won't help. Calling server.addBean(item) isn't a problem either.
~ David
Simonian, Marta M (US SSA) wrote
> Hi,
>
> We are using Solr 4.4 to ingest geo data and it's really slow. When we
> don't index the geo it takes seconds to ingest 100, 000 records but as
> soon as we add it takes 2 hours.
>
> Also we found that when changing the distErrPct from 0.025 to 0.1,
> 1000 rows are ingested in 20 sec vs 2 min. But we can't change that
> setting as we want our search to be as accurate as possible.
>
> About the environment we are running Solr on 6 CPUs and 8GB of memory.
> We've been monitoring the VMs and they seem to be ok.
>
> We are running on Tomcat but we might switch to Jetty to see if that
> will increase the performance.
>
> We use ConcurrentUpdateSolrServer(httpSolrServer, 5000, 2);
>
> We are saving a bean rather than a solr document (server.addBean(item)).
> I'm not sure if that could make it slow as it's going to do some
> conversion?
>
> Can you please let me know what are the best settings for Solr? Maybe
> some changes in the solrconfig.xml or the schema.xml?
> What are the preferred environment settings and resources?
>
> Thank you!
> Marta
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Ingesting-geo-data-into-Solr-very-slow-tp4081484p4081527.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Ingesting geo data into Solr very slow
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Hi Marta,
Presumably you are indexing polygons -- I suspect complex ones. There isn't
too much that you can do about this right now other than index them in
parallel. I see you are doing this in 2 threads; try 4, or maybe even 6.
Also, ensure that maxDistErr is reflective of the smallest distance you need
to distinguish between. It may help a little but not much. I can think of
some internal code details that might be improved but that doesn't help you
now.
There's some generic Solr things you can do to improve indexing performance
too like increasing the indexing buffer size (100MB -> 200MB) and the
mergeFactor (10->20 albeit temporarily and/or issue optimize), both in
solrconfig.xml.
Changing the servlet engine won't help. Calling server.addBean(item) isn't a
problem either.
~ David
Simonian, Marta M (US SSA) wrote
> Hi,
>
> We are using Solr 4.4 to ingest geo data and it's really slow. When we
> don't index the geo it takes seconds to ingest 100, 000 records but as
> soon as we add it takes 2 hours.
>
> Also we found that when changing the distErrPct from 0.025 to 0.1, 1000
> rows are ingested in 20 sec vs 2 min. But we can't change that setting as
> we want our search to be as accurate as possible.
>
> About the environment we are running Solr on 6 CPUs and 8GB of memory.
> We've been monitoring the VMs and they seem to be ok.
>
> We are running on Tomcat but we might switch to Jetty to see if that will
> increase the performance.
>
> We use ConcurrentUpdateSolrServer(httpSolrServer, 5000, 2);
>
> We are saving a bean rather than a solr document (server.addBean(item)).
> I'm not sure if that could make it slow as it's going to do some
> conversion?
>
> Can you please let me know what are the best settings for Solr? Maybe some
> changes in the solrconfig.xml or the schema.xml?
> What are the preferred environment settings and resources?
>
> Thank you!
> Marta
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/Ingesting-geo-data-into-Solr-very-slow-tp4081484p4081527.html
Sent from the Solr - User mailing list archive at Nabble.com.