You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by vivek sar <vi...@gmail.com> on 2009/04/10 22:48:27 UTC
Question on StreamingUpdateSolrServer
Hi,
I was using CommonsHttpSolrServer for indexing, but having two
threads writing (10K batches) at the same time was throwing,
"ProtocolException: Unbuffered entity enclosing request can not be repeated. "
I switched to StreamingUpdateSolrServer (using addBeans) and I don't
see the problem anymore. The speed is very fast - getting around
25k/sec (single thread), but I'm facing another problem. When the
indexer using StreamingUpdateSolrServer is running I'm not able to
send any url request from browser to Solr web app. I just get blank
page. I can't even get to the admin interface. I'm also not able to
shutdown the Tomcat running the Solr webapp when the Indexer is
running. I've to first stop the Indexer app and then stop the Tomcat.
I don't have this problem when using CommonsHttpSolrServer.
Here is how I'm creating it,
server = new StreamingUpdateSolrServer(url, 1000,3);
I simply call server.addBeans(...) on it. Is there anything else I
need to do to make use of StreamingUpdateSolrServer? Why does Tomcat
become unresponsive when Indexer using StreamingUpdateSolrServer is
running (though, indexing happens fine)?
Thanks,
-vivek
Re: Question on StreamingUpdateSolrServer
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Quick comment - why so shy with number of open file descriptors? On some nothing-special machines from several years ago I had this limit set to 30K+ - here, for example: http://www.simpy.com/user/otis :)
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: vivek sar <vi...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, April 14, 2009 3:12:41 AM
> Subject: Re: Question on StreamingUpdateSolrServer
>
> The machine's ulimit is set to 9000 and the OS has upper limit of
> 12000 on files. What would explain this? Has anyone tried Solr with 25
> cores on the same Solr instance?
>
> Thanks,
> -vivek
>
> 2009/4/13 Noble Paul നോബിള് नोब्ळ् :
> > On Tue, Apr 14, 2009 at 7:14 AM, vivek sar wrote:
> >> Some more update. As I mentioned earlier we are using multi-core Solr
> >> (up to 65 cores in one Solr instance with each core 10G). This was
> >> opening around 3000 file descriptors (lsof). I removed some cores and
> >> after some trial and error I found at 25 cores system seems to work
> >> fine (around 1400 file descriptors). Tomcat is responsive even when
> >> the indexing is happening at Solr (for 25 cores). But, as soon as it
> >> goes to 26 cores the Tomcat becomes unresponsive again. The puzzling
> >> thing is if I stop indexing I can search on even 65 cores, but while
> >> indexing is happening it seems to support only up to 25 cores.
> >>
> >> 1) Is there a limit on number of cores a Solr instance can handle?
> >> 2) Does Solr do anything to the existing cores while indexing? I'm
> >> writing to only one core at a time.
> > There is no hard limit (it is Integer.MAX_VALUE) . But inreality your
> > mileage depends on your hardware and no:of file handles the OS can
> > open
> >>
> >> We are struggling to find why Tomcat stops responding on high number
> >> of cores while indexing is in-progress. Any help is very much
> >> appreciated.
> >>
> >> Thanks,
> >> -vivek
> >>
> >> On Mon, Apr 13, 2009 at 10:52 AM, vivek sar wrote:
> >>> Here is some more information about my setup,
> >>>
> >>> Solr - v1.4 (nightly build 03/29/09)
> >>> Servlet Container - Tomcat 6.0.18
> >>> JVM - 1.6.0 (64 bit)
> >>> OS - Mac OS X Server 10.5.6
> >>>
> >>> Hardware Overview:
> >>>
> >>> Processor Name: Quad-Core Intel Xeon
> >>> Processor Speed: 3 GHz
> >>> Number Of Processors: 2
> >>> Total Number Of Cores: 8
> >>> L2 Cache (per processor): 12 MB
> >>> Memory: 20 GB
> >>> Bus Speed: 1.6 GHz
> >>>
> >>> JVM Parameters (for Solr):
> >>>
> >>> export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
> >>> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
> >>> -Dsun.rmi.dgc.client.gcInterval=3600000
> >>> -Dsun.rmi.dgc.server.gcInterval=3600000"
> >>>
> >>> Other:
> >>>
> >>> lsof|grep solr|wc -l
> >>> 2493
> >>>
> >>> ulimit -an
> >>> open files (-n) 9000
> >>>
> >>> Tomcat
> >>>
> >>> connectionTimeout="20000"
> >>> maxThreads="100" />
> >>>
> >>> Total Solr cores on same instance - 65
> >>>
> >>> useCompoundFile - true
> >>>
> >>> The tests I ran,
> >>>
> >>> While Indexer is running
> >>> 1) Go to "http://juum19.co.com:8080/solr" - returns blank page (no
> >>> error in the catalina.out)
> >>>
> >>> 2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
> >>> by foreign host"
> >>>
> >>> Stop the Indexer Program (Tomcat is still running with Solr)
> >>>
> >>> 3) Go to "http://juum19.co.com:8080/solr" - works ok, shows the list
> >>> of all the Solr cores
> >>>
> >>> 4) Try telnet - able to Telnet fine
> >>>
> >>> 5) Now comment out all the caches in solrconfig.xml. Try same tests,
> >>> but the Tomcat still doesn't response.
> >>>
> >>> Is there a way to stop the auto-warmer. I commented out the caches in
> >>> the solrconfig.xml but still see the following log,
> >>>
> >>> INFO: autowarming result for Searcher@3aba3830 main
> >>>
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >>>
> >>> INFO: Closing Searcher@175dc1e2
> >>> main
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >>>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >>>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >>>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> >>>
> >>>
> >>> 6) Change the Indexer frequency so it runs every 2 min (instead of all
> >>> the time). I noticed once the commit is done, I'm able to run my
> >>> searches. During commit and auto-warming period I just get blank page.
> >>>
> >>> 7) Changed from Solrj to XML update - I still get the blank page
> >>> whenever update/commit is happening.
> >>>
> >>> Apr 13, 2009 6:46:18 PM
> >>> org.apache.solr.update.processor.LogUpdateProcessor finish
> >>> INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
> >>> 621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
> >>> Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
> >>> INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
> >>>
> >>>
> >>> So, looks like it's not just StreamingUpdateSolrServer, but whenever
> >>> the update/commit is happening I'm not able to search. I don't know if
> >>> it's related to using multi-core. In this test I was using only single
> >>> thread for update to a single core using only single Solr instance.
> >>>
> >>> So, it's clearly related to index process (update, commit and
> >>> auto-warming). As soon as update/commit/auto-warming is completed I'm
> >>> able to run my queries again. Is there anything that could stop
> >>> searching while update process is in-progress - like any lock or
> >>> something?
> >>>
> >>> Any other ideas?
> >>>
> >>> Thanks,
> >>> -vivek
> >>>
> >>> On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
> >>> wrote:
> >>>> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar wrote:
> >>>>
> >>>>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
> >>>>> there any limitation that I can't search during commit or
> >>>>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
> >>>>> top) - so it's unlikely that the CPU was pegged.
> >>>>>
> >>>>>
> >>>> No, there is no such limitation. The old searcher will continue to serve
> >>>> search requests until the new one is warmed and registered.
> >>>>
> >>>> So, CPU does not seem to be an issue. Does this happen only when you use
> >>>> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
> >>>> you using? Which servlet container and version?
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > --Noble Paul
> >
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
The machine's ulimit is set to 9000 and the OS has upper limit of
12000 on files. What would explain this? Has anyone tried Solr with 25
cores on the same Solr instance?
Thanks,
-vivek
2009/4/13 Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>:
> On Tue, Apr 14, 2009 at 7:14 AM, vivek sar <vi...@gmail.com> wrote:
>> Some more update. As I mentioned earlier we are using multi-core Solr
>> (up to 65 cores in one Solr instance with each core 10G). This was
>> opening around 3000 file descriptors (lsof). I removed some cores and
>> after some trial and error I found at 25 cores system seems to work
>> fine (around 1400 file descriptors). Tomcat is responsive even when
>> the indexing is happening at Solr (for 25 cores). But, as soon as it
>> goes to 26 cores the Tomcat becomes unresponsive again. The puzzling
>> thing is if I stop indexing I can search on even 65 cores, but while
>> indexing is happening it seems to support only up to 25 cores.
>>
>> 1) Is there a limit on number of cores a Solr instance can handle?
>> 2) Does Solr do anything to the existing cores while indexing? I'm
>> writing to only one core at a time.
> There is no hard limit (it is Integer.MAX_VALUE) . But inreality your
> mileage depends on your hardware and no:of file handles the OS can
> open
>>
>> We are struggling to find why Tomcat stops responding on high number
>> of cores while indexing is in-progress. Any help is very much
>> appreciated.
>>
>> Thanks,
>> -vivek
>>
>> On Mon, Apr 13, 2009 at 10:52 AM, vivek sar <vi...@gmail.com> wrote:
>>> Here is some more information about my setup,
>>>
>>> Solr - v1.4 (nightly build 03/29/09)
>>> Servlet Container - Tomcat 6.0.18
>>> JVM - 1.6.0 (64 bit)
>>> OS - Mac OS X Server 10.5.6
>>>
>>> Hardware Overview:
>>>
>>> Processor Name: Quad-Core Intel Xeon
>>> Processor Speed: 3 GHz
>>> Number Of Processors: 2
>>> Total Number Of Cores: 8
>>> L2 Cache (per processor): 12 MB
>>> Memory: 20 GB
>>> Bus Speed: 1.6 GHz
>>>
>>> JVM Parameters (for Solr):
>>>
>>> export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
>>> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
>>> -Dsun.rmi.dgc.client.gcInterval=3600000
>>> -Dsun.rmi.dgc.server.gcInterval=3600000"
>>>
>>> Other:
>>>
>>> lsof|grep solr|wc -l
>>> 2493
>>>
>>> ulimit -an
>>> open files (-n) 9000
>>>
>>> Tomcat
>>> <Connector port="8080" protocol="HTTP/1.1"
>>> connectionTimeout="20000"
>>> maxThreads="100" />
>>>
>>> Total Solr cores on same instance - 65
>>>
>>> useCompoundFile - true
>>>
>>> The tests I ran,
>>>
>>> While Indexer is running
>>> 1) Go to "http://juum19.co.com:8080/solr" - returns blank page (no
>>> error in the catalina.out)
>>>
>>> 2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
>>> by foreign host"
>>>
>>> Stop the Indexer Program (Tomcat is still running with Solr)
>>>
>>> 3) Go to "http://juum19.co.com:8080/solr" - works ok, shows the list
>>> of all the Solr cores
>>>
>>> 4) Try telnet - able to Telnet fine
>>>
>>> 5) Now comment out all the caches in solrconfig.xml. Try same tests,
>>> but the Tomcat still doesn't response.
>>>
>>> Is there a way to stop the auto-warmer. I commented out the caches in
>>> the solrconfig.xml but still see the following log,
>>>
>>> INFO: autowarming result for Searcher@3aba3830 main
>>> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>>
>>> INFO: Closing Searcher@175dc1e2
>>> main fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>>
>>>
>>> 6) Change the Indexer frequency so it runs every 2 min (instead of all
>>> the time). I noticed once the commit is done, I'm able to run my
>>> searches. During commit and auto-warming period I just get blank page.
>>>
>>> 7) Changed from Solrj to XML update - I still get the blank page
>>> whenever update/commit is happening.
>>>
>>> Apr 13, 2009 6:46:18 PM
>>> org.apache.solr.update.processor.LogUpdateProcessor finish
>>> INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
>>> 621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
>>> Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
>>> INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
>>>
>>>
>>> So, looks like it's not just StreamingUpdateSolrServer, but whenever
>>> the update/commit is happening I'm not able to search. I don't know if
>>> it's related to using multi-core. In this test I was using only single
>>> thread for update to a single core using only single Solr instance.
>>>
>>> So, it's clearly related to index process (update, commit and
>>> auto-warming). As soon as update/commit/auto-warming is completed I'm
>>> able to run my queries again. Is there anything that could stop
>>> searching while update process is in-progress - like any lock or
>>> something?
>>>
>>> Any other ideas?
>>>
>>> Thanks,
>>> -vivek
>>>
>>> On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
>>> <sh...@gmail.com> wrote:
>>>> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vi...@gmail.com> wrote:
>>>>
>>>>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
>>>>> there any limitation that I can't search during commit or
>>>>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
>>>>> top) - so it's unlikely that the CPU was pegged.
>>>>>
>>>>>
>>>> No, there is no such limitation. The old searcher will continue to serve
>>>> search requests until the new one is warmed and registered.
>>>>
>>>> So, CPU does not seem to be an issue. Does this happen only when you use
>>>> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
>>>> you using? Which servlet container and version?
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>
>
>
>
> --
> --Noble Paul
>
Re: Question on StreamingUpdateSolrServer
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
On Tue, Apr 14, 2009 at 7:14 AM, vivek sar <vi...@gmail.com> wrote:
> Some more update. As I mentioned earlier we are using multi-core Solr
> (up to 65 cores in one Solr instance with each core 10G). This was
> opening around 3000 file descriptors (lsof). I removed some cores and
> after some trial and error I found at 25 cores system seems to work
> fine (around 1400 file descriptors). Tomcat is responsive even when
> the indexing is happening at Solr (for 25 cores). But, as soon as it
> goes to 26 cores the Tomcat becomes unresponsive again. The puzzling
> thing is if I stop indexing I can search on even 65 cores, but while
> indexing is happening it seems to support only up to 25 cores.
>
> 1) Is there a limit on number of cores a Solr instance can handle?
> 2) Does Solr do anything to the existing cores while indexing? I'm
> writing to only one core at a time.
There is no hard limit (it is Integer.MAX_VALUE) . But inreality your
mileage depends on your hardware and no:of file handles the OS can
open
>
> We are struggling to find why Tomcat stops responding on high number
> of cores while indexing is in-progress. Any help is very much
> appreciated.
>
> Thanks,
> -vivek
>
> On Mon, Apr 13, 2009 at 10:52 AM, vivek sar <vi...@gmail.com> wrote:
>> Here is some more information about my setup,
>>
>> Solr - v1.4 (nightly build 03/29/09)
>> Servlet Container - Tomcat 6.0.18
>> JVM - 1.6.0 (64 bit)
>> OS - Mac OS X Server 10.5.6
>>
>> Hardware Overview:
>>
>> Processor Name: Quad-Core Intel Xeon
>> Processor Speed: 3 GHz
>> Number Of Processors: 2
>> Total Number Of Cores: 8
>> L2 Cache (per processor): 12 MB
>> Memory: 20 GB
>> Bus Speed: 1.6 GHz
>>
>> JVM Parameters (for Solr):
>>
>> export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
>> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
>> -Dsun.rmi.dgc.client.gcInterval=3600000
>> -Dsun.rmi.dgc.server.gcInterval=3600000"
>>
>> Other:
>>
>> lsof|grep solr|wc -l
>> 2493
>>
>> ulimit -an
>> open files (-n) 9000
>>
>> Tomcat
>> <Connector port="8080" protocol="HTTP/1.1"
>> connectionTimeout="20000"
>> maxThreads="100" />
>>
>> Total Solr cores on same instance - 65
>>
>> useCompoundFile - true
>>
>> The tests I ran,
>>
>> While Indexer is running
>> 1) Go to "http://juum19.co.com:8080/solr" - returns blank page (no
>> error in the catalina.out)
>>
>> 2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
>> by foreign host"
>>
>> Stop the Indexer Program (Tomcat is still running with Solr)
>>
>> 3) Go to "http://juum19.co.com:8080/solr" - works ok, shows the list
>> of all the Solr cores
>>
>> 4) Try telnet - able to Telnet fine
>>
>> 5) Now comment out all the caches in solrconfig.xml. Try same tests,
>> but the Tomcat still doesn't response.
>>
>> Is there a way to stop the auto-warmer. I commented out the caches in
>> the solrconfig.xml but still see the following log,
>>
>> INFO: autowarming result for Searcher@3aba3830 main
>> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>
>> INFO: Closing Searcher@175dc1e2
>> main fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>
>>
>> 6) Change the Indexer frequency so it runs every 2 min (instead of all
>> the time). I noticed once the commit is done, I'm able to run my
>> searches. During commit and auto-warming period I just get blank page.
>>
>> 7) Changed from Solrj to XML update - I still get the blank page
>> whenever update/commit is happening.
>>
>> Apr 13, 2009 6:46:18 PM
>> org.apache.solr.update.processor.LogUpdateProcessor finish
>> INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
>> 621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
>> Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
>> INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
>>
>>
>> So, looks like it's not just StreamingUpdateSolrServer, but whenever
>> the update/commit is happening I'm not able to search. I don't know if
>> it's related to using multi-core. In this test I was using only single
>> thread for update to a single core using only single Solr instance.
>>
>> So, it's clearly related to index process (update, commit and
>> auto-warming). As soon as update/commit/auto-warming is completed I'm
>> able to run my queries again. Is there anything that could stop
>> searching while update process is in-progress - like any lock or
>> something?
>>
>> Any other ideas?
>>
>> Thanks,
>> -vivek
>>
>> On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
>> <sh...@gmail.com> wrote:
>>> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vi...@gmail.com> wrote:
>>>
>>>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
>>>> there any limitation that I can't search during commit or
>>>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
>>>> top) - so it's unlikely that the CPU was pegged.
>>>>
>>>>
>>> No, there is no such limitation. The old searcher will continue to serve
>>> search requests until the new one is warmed and registered.
>>>
>>> So, CPU does not seem to be an issue. Does this happen only when you use
>>> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
>>> you using? Which servlet container and version?
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>
--
--Noble Paul
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
Some more update. As I mentioned earlier we are using multi-core Solr
(up to 65 cores in one Solr instance with each core 10G). This was
opening around 3000 file descriptors (lsof). I removed some cores and
after some trial and error I found at 25 cores system seems to work
fine (around 1400 file descriptors). Tomcat is responsive even when
the indexing is happening at Solr (for 25 cores). But, as soon as it
goes to 26 cores the Tomcat becomes unresponsive again. The puzzling
thing is if I stop indexing I can search on even 65 cores, but while
indexing is happening it seems to support only up to 25 cores.
1) Is there a limit on number of cores a Solr instance can handle?
2) Does Solr do anything to the existing cores while indexing? I'm
writing to only one core at a time.
We are struggling to find why Tomcat stops responding on high number
of cores while indexing is in-progress. Any help is very much
appreciated.
Thanks,
-vivek
On Mon, Apr 13, 2009 at 10:52 AM, vivek sar <vi...@gmail.com> wrote:
> Here is some more information about my setup,
>
> Solr - v1.4 (nightly build 03/29/09)
> Servlet Container - Tomcat 6.0.18
> JVM - 1.6.0 (64 bit)
> OS - Mac OS X Server 10.5.6
>
> Hardware Overview:
>
> Processor Name: Quad-Core Intel Xeon
> Processor Speed: 3 GHz
> Number Of Processors: 2
> Total Number Of Cores: 8
> L2 Cache (per processor): 12 MB
> Memory: 20 GB
> Bus Speed: 1.6 GHz
>
> JVM Parameters (for Solr):
>
> export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
> -Dsun.rmi.dgc.client.gcInterval=3600000
> -Dsun.rmi.dgc.server.gcInterval=3600000"
>
> Other:
>
> lsof|grep solr|wc -l
> 2493
>
> ulimit -an
> open files (-n) 9000
>
> Tomcat
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> maxThreads="100" />
>
> Total Solr cores on same instance - 65
>
> useCompoundFile - true
>
> The tests I ran,
>
> While Indexer is running
> 1) Go to "http://juum19.co.com:8080/solr" - returns blank page (no
> error in the catalina.out)
>
> 2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
> by foreign host"
>
> Stop the Indexer Program (Tomcat is still running with Solr)
>
> 3) Go to "http://juum19.co.com:8080/solr" - works ok, shows the list
> of all the Solr cores
>
> 4) Try telnet - able to Telnet fine
>
> 5) Now comment out all the caches in solrconfig.xml. Try same tests,
> but the Tomcat still doesn't response.
>
> Is there a way to stop the auto-warmer. I commented out the caches in
> the solrconfig.xml but still see the following log,
>
> INFO: autowarming result for Searcher@3aba3830 main
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>
> INFO: Closing Searcher@175dc1e2
> main fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>
>
> 6) Change the Indexer frequency so it runs every 2 min (instead of all
> the time). I noticed once the commit is done, I'm able to run my
> searches. During commit and auto-warming period I just get blank page.
>
> 7) Changed from Solrj to XML update - I still get the blank page
> whenever update/commit is happening.
>
> Apr 13, 2009 6:46:18 PM
> org.apache.solr.update.processor.LogUpdateProcessor finish
> INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
> 621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
> Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
> INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
>
>
> So, looks like it's not just StreamingUpdateSolrServer, but whenever
> the update/commit is happening I'm not able to search. I don't know if
> it's related to using multi-core. In this test I was using only single
> thread for update to a single core using only single Solr instance.
>
> So, it's clearly related to index process (update, commit and
> auto-warming). As soon as update/commit/auto-warming is completed I'm
> able to run my queries again. Is there anything that could stop
> searching while update process is in-progress - like any lock or
> something?
>
> Any other ideas?
>
> Thanks,
> -vivek
>
> On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vi...@gmail.com> wrote:
>>
>>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
>>> there any limitation that I can't search during commit or
>>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
>>> top) - so it's unlikely that the CPU was pegged.
>>>
>>>
>> No, there is no such limitation. The old searcher will continue to serve
>> search requests until the new one is warmed and registered.
>>
>> So, CPU does not seem to be an issue. Does this happen only when you use
>> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
>> you using? Which servlet container and version?
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
Here is some more information about my setup,
Solr - v1.4 (nightly build 03/29/09)
Servlet Container - Tomcat 6.0.18
JVM - 1.6.0 (64 bit)
OS - Mac OS X Server 10.5.6
Hardware Overview:
Processor Name: Quad-Core Intel Xeon
Processor Speed: 3 GHz
Number Of Processors: 2
Total Number Of Cores: 8
L2 Cache (per processor): 12 MB
Memory: 20 GB
Bus Speed: 1.6 GHz
JVM Parameters (for Solr):
export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000"
Other:
lsof|grep solr|wc -l
2493
ulimit -an
open files (-n) 9000
Tomcat
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
maxThreads="100" />
Total Solr cores on same instance - 65
useCompoundFile - true
The tests I ran,
While Indexer is running
1) Go to "http://juum19.co.com:8080/solr" - returns blank page (no
error in the catalina.out)
2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
by foreign host"
Stop the Indexer Program (Tomcat is still running with Solr)
3) Go to "http://juum19.co.com:8080/solr" - works ok, shows the list
of all the Solr cores
4) Try telnet - able to Telnet fine
5) Now comment out all the caches in solrconfig.xml. Try same tests,
but the Tomcat still doesn't response.
Is there a way to stop the auto-warmer. I commented out the caches in
the solrconfig.xml but still see the following log,
INFO: autowarming result for Searcher@3aba3830 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
INFO: Closing Searcher@175dc1e2
main fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
6) Change the Indexer frequency so it runs every 2 min (instead of all
the time). I noticed once the commit is done, I'm able to run my
searches. During commit and auto-warming period I just get blank page.
7) Changed from Solrj to XML update - I still get the blank page
whenever update/commit is happening.
Apr 13, 2009 6:46:18 PM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
So, looks like it's not just StreamingUpdateSolrServer, but whenever
the update/commit is happening I'm not able to search. I don't know if
it's related to using multi-core. In this test I was using only single
thread for update to a single core using only single Solr instance.
So, it's clearly related to index process (update, commit and
auto-warming). As soon as update/commit/auto-warming is completed I'm
able to run my queries again. Is there anything that could stop
searching while update process is in-progress - like any lock or
something?
Any other ideas?
Thanks,
-vivek
On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vi...@gmail.com> wrote:
>
>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
>> there any limitation that I can't search during commit or
>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
>> top) - so it's unlikely that the CPU was pegged.
>>
>>
> No, there is no such limitation. The old searcher will continue to serve
> search requests until the new one is warmed and registered.
>
> So, CPU does not seem to be an issue. Does this happen only when you use
> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
> you using? Which servlet container and version?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Question on StreamingUpdateSolrServer
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vi...@gmail.com> wrote:
> I index in 10K batches and commit after 5 index cyles (after 50K). Is
> there any limitation that I can't search during commit or
> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
> top) - so it's unlikely that the CPU was pegged.
>
>
No, there is no such limitation. The old searcher will continue to serve
search requests until the new one is warmed and registered.
So, CPU does not seem to be an issue. Does this happen only when you use
StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
you using? Which servlet container and version?
--
Regards,
Shalin Shekhar Mangar.
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
I index in 10K batches and commit after 5 index cyles (after 50K). Is
there any limitation that I can't search during commit or
auto-warming? I got 8 CPU cores and only 2 were showing busy (using
top) - so it's unlikely that the CPU was pegged.
2009/4/12 Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>:
> If you use StreamingUpdateSolrServer it POSTs all the docs in a single
> request. 10 million docs may be a bit too much for a single request. I
> guess you should batch it in multiple requests of smaller chunks,
>
> It is likely that the CPU is really hot when the autowarming is hapening.
>
> getting a decent search perf w/o autowarming is not easy .
>
> autowarmCount is an attribute of a cache .see here
> http://wiki.apache.org/solr/SolrCaching
>
> On Mon, Apr 13, 2009 at 3:32 AM, vivek sar <vi...@gmail.com> wrote:
>> Thanks Shalin.
>>
>> I noticed couple more things. As I index around 100 million records a
>> day, my Indexer is running pretty much at all times throughout the
>> day. Whenever I run a search query I usually get "connection reset"
>> when the commit is happening and get "blank page" when the
>> auto-warming of searchers is happening. Here are my questions,
>>
>> 1) Is this coincidence or a known issue? Can't we search while commit
>> or auto-warming is happening?
>> 2) How do I stop auto-warming? My search traffic is very low so I'm
>> trying to turn off auto-warming after commit has happened - is there
>> anything in the solrconfig.xml to do that?
>> 3) What would be the best strategy for searching in my scenario where
>> commits may be happening all the time (I commit every 50K records - so
>> every 30-60 sec there is a commit happening followed by auto-warming
>> that takes 40 sec)?
>>
>> Search frequency is pretty low for us, but we want to make sure that
>> whenever it happens it is fast enough and returns result (instead of
>> exception or a blank screen).
>>
>> Thanks for all the help.
>>
>> -vivek
>>
>>
>>
>> On Sat, Apr 11, 2009 at 1:48 PM, Shalin Shekhar Mangar
>> <sh...@gmail.com> wrote:
>>> On Sun, Apr 12, 2009 at 2:15 AM, vivek sar <vi...@gmail.com> wrote:
>>>
>>>>
>>>> The problem is I don't see any error message in the catalina.out. I
>>>> don't even see the request coming in - I simply get blank page on
>>>> browser. If I keep trying the request goes through and I get respond
>>>> from Solr, but then it become unresponsive again or sometimes throws
>>>> "connection reset" error. I'm not sure why would it work sometimes and
>>>> not the other times for the same query. As soon as I stop the Indexer
>>>> process things start working fine. Any way I can debug this problem?
>>>>
>>>
>>> I'm not sure. I've never seen this issue myself.
>>>
>>> Could you try using the bundled jetty instead of Tomcat or on a different
>>> box just to make sure this is not an environment specific issue?
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>
>
>
> --
> --Noble Paul
>
Re: Question on StreamingUpdateSolrServer
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
If you use StreamingUpdateSolrServer it POSTs all the docs in a single
request. 10 million docs may be a bit too much for a single request. I
guess you should batch it in multiple requests of smaller chunks,
It is likely that the CPU is really hot when the autowarming is hapening.
getting a decent search perf w/o autowarming is not easy .
autowarmCount is an attribute of a cache .see here
http://wiki.apache.org/solr/SolrCaching
On Mon, Apr 13, 2009 at 3:32 AM, vivek sar <vi...@gmail.com> wrote:
> Thanks Shalin.
>
> I noticed couple more things. As I index around 100 million records a
> day, my Indexer is running pretty much at all times throughout the
> day. Whenever I run a search query I usually get "connection reset"
> when the commit is happening and get "blank page" when the
> auto-warming of searchers is happening. Here are my questions,
>
> 1) Is this coincidence or a known issue? Can't we search while commit
> or auto-warming is happening?
> 2) How do I stop auto-warming? My search traffic is very low so I'm
> trying to turn off auto-warming after commit has happened - is there
> anything in the solrconfig.xml to do that?
> 3) What would be the best strategy for searching in my scenario where
> commits may be happening all the time (I commit every 50K records - so
> every 30-60 sec there is a commit happening followed by auto-warming
> that takes 40 sec)?
>
> Search frequency is pretty low for us, but we want to make sure that
> whenever it happens it is fast enough and returns result (instead of
> exception or a blank screen).
>
> Thanks for all the help.
>
> -vivek
>
>
>
> On Sat, Apr 11, 2009 at 1:48 PM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>> On Sun, Apr 12, 2009 at 2:15 AM, vivek sar <vi...@gmail.com> wrote:
>>
>>>
>>> The problem is I don't see any error message in the catalina.out. I
>>> don't even see the request coming in - I simply get blank page on
>>> browser. If I keep trying the request goes through and I get respond
>>> from Solr, but then it become unresponsive again or sometimes throws
>>> "connection reset" error. I'm not sure why would it work sometimes and
>>> not the other times for the same query. As soon as I stop the Indexer
>>> process things start working fine. Any way I can debug this problem?
>>>
>>
>> I'm not sure. I've never seen this issue myself.
>>
>> Could you try using the bundled jetty instead of Tomcat or on a different
>> box just to make sure this is not an environment specific issue?
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
--
--Noble Paul
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
Thanks Shalin.
I noticed couple more things. As I index around 100 million records a
day, my Indexer is running pretty much at all times throughout the
day. Whenever I run a search query I usually get "connection reset"
when the commit is happening and get "blank page" when the
auto-warming of searchers is happening. Here are my questions,
1) Is this coincidence or a known issue? Can't we search while commit
or auto-warming is happening?
2) How do I stop auto-warming? My search traffic is very low so I'm
trying to turn off auto-warming after commit has happened - is there
anything in the solrconfig.xml to do that?
3) What would be the best strategy for searching in my scenario where
commits may be happening all the time (I commit every 50K records - so
every 30-60 sec there is a commit happening followed by auto-warming
that takes 40 sec)?
Search frequency is pretty low for us, but we want to make sure that
whenever it happens it is fast enough and returns result (instead of
exception or a blank screen).
Thanks for all the help.
-vivek
On Sat, Apr 11, 2009 at 1:48 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Sun, Apr 12, 2009 at 2:15 AM, vivek sar <vi...@gmail.com> wrote:
>
>>
>> The problem is I don't see any error message in the catalina.out. I
>> don't even see the request coming in - I simply get blank page on
>> browser. If I keep trying the request goes through and I get respond
>> from Solr, but then it become unresponsive again or sometimes throws
>> "connection reset" error. I'm not sure why would it work sometimes and
>> not the other times for the same query. As soon as I stop the Indexer
>> process things start working fine. Any way I can debug this problem?
>>
>
> I'm not sure. I've never seen this issue myself.
>
> Could you try using the bundled jetty instead of Tomcat or on a different
> box just to make sure this is not an environment specific issue?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Question on StreamingUpdateSolrServer
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Sun, Apr 12, 2009 at 2:15 AM, vivek sar <vi...@gmail.com> wrote:
>
> The problem is I don't see any error message in the catalina.out. I
> don't even see the request coming in - I simply get blank page on
> browser. If I keep trying the request goes through and I get respond
> from Solr, but then it become unresponsive again or sometimes throws
> "connection reset" error. I'm not sure why would it work sometimes and
> not the other times for the same query. As soon as I stop the Indexer
> process things start working fine. Any way I can debug this problem?
>
I'm not sure. I've never seen this issue myself.
Could you try using the bundled jetty instead of Tomcat or on a different
box just to make sure this is not an environment specific issue?
--
Regards,
Shalin Shekhar Mangar.
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
Thanks Shalin.
The problem is I don't see any error message in the catalina.out. I
don't even see the request coming in - I simply get blank page on
browser. If I keep trying the request goes through and I get respond
from Solr, but then it become unresponsive again or sometimes throws
"connection reset" error. I'm not sure why would it work sometimes and
not the other times for the same query. As soon as I stop the Indexer
process things start working fine. Any way I can debug this problem?
-vivek
On Fri, Apr 10, 2009 at 11:05 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Sat, Apr 11, 2009 at 3:29 AM, vivek sar <vi...@gmail.com> wrote:
>
>> I also noticed that the Solr app has over 6000 file handles open -
>>
>> "lsof | grep solr | wc -l" - shows 6455
>>
>> I've 10 cores (using multi-core) managed by the same Solr instance. As
>> soon as start up the Tomcat the open file count goes up to 6400. Few
>> questions,
>>
>> 1) Why is Solr holding on to all the segments from all the cores - is
>> it because of auto-warmer?
>
>
> You have 10 cores, so Solr opens 10 indexes, each of which contains multiple
> files. That is one reason. Apart from that, Tomcat will keep some file
> handles for incoming connections.
>
>
>>
>> 2) How can I reduce the open file count?
>
>
> Are they causing a problem? Tomcat will log messages when it cannot accept
> incoming connections if it runs out of available file handles. But if you
> experiencing issues, you can increase the file handle limit or you can set
> useCompoundFile=true in solrconfig.xml.
>
>
>>
>> 3) Is there a way to stop the auto-warmer?
>> 4) Could this be related to "Tomcat returning blank page for every
>> request"?
>>
>
> It could be. Check the Tomcat and Solr logs.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Question on StreamingUpdateSolrServer
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Sat, Apr 11, 2009 at 3:29 AM, vivek sar <vi...@gmail.com> wrote:
> I also noticed that the Solr app has over 6000 file handles open -
>
> "lsof | grep solr | wc -l" - shows 6455
>
> I've 10 cores (using multi-core) managed by the same Solr instance. As
> soon as start up the Tomcat the open file count goes up to 6400. Few
> questions,
>
> 1) Why is Solr holding on to all the segments from all the cores - is
> it because of auto-warmer?
You have 10 cores, so Solr opens 10 indexes, each of which contains multiple
files. That is one reason. Apart from that, Tomcat will keep some file
handles for incoming connections.
>
> 2) How can I reduce the open file count?
Are they causing a problem? Tomcat will log messages when it cannot accept
incoming connections if it runs out of available file handles. But if you
experiencing issues, you can increase the file handle limit or you can set
useCompoundFile=true in solrconfig.xml.
>
> 3) Is there a way to stop the auto-warmer?
> 4) Could this be related to "Tomcat returning blank page for every
> request"?
>
It could be. Check the Tomcat and Solr logs.
--
Regards,
Shalin Shekhar Mangar.
Re: Question on StreamingUpdateSolrServer
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Apr 15, 2009 at 7:28 PM, vivek sar <vi...@gmail.com> wrote:
> lsof at
> this point usually shows at 1400, but my ulimit is much higher than
> that.
Could you be hitting a kernel limit?
cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr
http://www.netadmintools.com/art295.html
-Yonik
http://www.lucidimagination.com
Re: Question on StreamingUpdateSolrServer
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Lots of little things to look at here.
You should do lsof as root, and it looks like you aren't doing that.
You should double-check Tomcat's maxThreads param in server.xml.
You should give Jetty a try.
I don't think you said anything about looking at the container's or solr logs and finding errors.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: vivek sar <vi...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, April 15, 2009 7:28:57 PM
> Subject: Re: Question on StreamingUpdateSolrServer
>
> Thanks Otis.
>
> I did increase the number of file descriptors to 22K, but I still get
> this problem. I've noticed following so far,
>
> 1) As soon as I get to around 1140 index segments (this is total over
> multiple cores) I start seeing this problem.
> 2) When the problem starts occassionally the index request
> (solrserver.commit) also fails with the following error,
> java.net.SocketException: Connection reset
> 3) Whenever the commit fails, I'm able to access Solr by the browser
> (http://ets11.co.com/solr). If the commit is succssfull and going on I
> get blank page on Firefox. Even the telnet to 8080 fails with
> "Connection closed by foreign host."
>
> It does seem like there is some resource issue as it happens only once
> we reach a breaking point (too many index segment files) - lsof at
> this point usually shows at 1400, but my ulimit is much higher than
> that.
>
> I already use compound format for index files. I can also run optimize
> occassionally (though not preferred as it blocks the whole index cycle
> for a long time). I do want to find out what resource limitation is
> causing this and it has to do something with when Indexer is
> committing the records where there are large number of segment files.
>
> Any other ideas?
>
> Thanks,
> -vivek
>
> On Wed, Apr 15, 2009 at 3:10 PM, Otis Gospodnetic
> wrote:
> >
> > One more thing. I don't think this was mentioned, but you can:
> > - optimize your indices
> > - use compound index format
> >
> > That will lower the number of open file handles.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: vivek sar
> >> To: solr-user@lucene.apache.org
> >> Sent: Friday, April 10, 2009 5:59:37 PM
> >> Subject: Re: Question on StreamingUpdateSolrServer
> >>
> >> I also noticed that the Solr app has over 6000 file handles open -
> >>
> >> "lsof | grep solr | wc -l" - shows 6455
> >>
> >> I've 10 cores (using multi-core) managed by the same Solr instance. As
> >> soon as start up the Tomcat the open file count goes up to 6400. Few
> >> questions,
> >>
> >> 1) Why is Solr holding on to all the segments from all the cores - is
> >> it because of auto-warmer?
> >> 2) How can I reduce the open file count?
> >> 3) Is there a way to stop the auto-warmer?
> >> 4) Could this be related to "Tomcat returning blank page for every request"?
> >>
> >> Any ideas?
> >>
> >> Thanks,
> >> -vivek
> >>
> >> On Fri, Apr 10, 2009 at 1:48 PM, vivek sar wrote:
> >> > Hi,
> >> >
> >> > I was using CommonsHttpSolrServer for indexing, but having two
> >> > threads writing (10K batches) at the same time was throwing,
> >> >
> >> > "ProtocolException: Unbuffered entity enclosing request can not be
> repeated.
> >> "
> >> >
> >> > I switched to StreamingUpdateSolrServer (using addBeans) and I don't
> >> > see the problem anymore. The speed is very fast - getting around
> >> > 25k/sec (single thread), but I'm facing another problem. When the
> >> > indexer using StreamingUpdateSolrServer is running I'm not able to
> >> > send any url request from browser to Solr web app. I just get blank
> >> > page. I can't even get to the admin interface. I'm also not able to
> >> > shutdown the Tomcat running the Solr webapp when the Indexer is
> >> > running. I've to first stop the Indexer app and then stop the Tomcat.
> >> > I don't have this problem when using CommonsHttpSolrServer.
> >> >
> >> > Here is how I'm creating it,
> >> >
> >> > server = new StreamingUpdateSolrServer(url, 1000,3);
> >> >
> >> > I simply call server.addBeans(...) on it. Is there anything else I
> >> > need to do to make use of StreamingUpdateSolrServer? Why does Tomcat
> >> > become unresponsive when Indexer using StreamingUpdateSolrServer is
> >> > running (though, indexing happens fine)?
> >> >
> >> > Thanks,
> >> > -vivek
> >> >
> >
> >
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
Thanks Otis.
I did increase the number of file descriptors to 22K, but I still get
this problem. I've noticed following so far,
1) As soon as I get to around 1140 index segments (this is total over
multiple cores) I start seeing this problem.
2) When the problem starts occassionally the index request
(solrserver.commit) also fails with the following error,
java.net.SocketException: Connection reset
3) Whenever the commit fails, I'm able to access Solr by the browser
(http://ets11.co.com/solr). If the commit is succssfull and going on I
get blank page on Firefox. Even the telnet to 8080 fails with
"Connection closed by foreign host."
It does seem like there is some resource issue as it happens only once
we reach a breaking point (too many index segment files) - lsof at
this point usually shows at 1400, but my ulimit is much higher than
that.
I already use compound format for index files. I can also run optimize
occassionally (though not preferred as it blocks the whole index cycle
for a long time). I do want to find out what resource limitation is
causing this and it has to do something with when Indexer is
committing the records where there are large number of segment files.
Any other ideas?
Thanks,
-vivek
On Wed, Apr 15, 2009 at 3:10 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> One more thing. I don't think this was mentioned, but you can:
> - optimize your indices
> - use compound index format
>
> That will lower the number of open file handles.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: vivek sar <vi...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Friday, April 10, 2009 5:59:37 PM
>> Subject: Re: Question on StreamingUpdateSolrServer
>>
>> I also noticed that the Solr app has over 6000 file handles open -
>>
>> "lsof | grep solr | wc -l" - shows 6455
>>
>> I've 10 cores (using multi-core) managed by the same Solr instance. As
>> soon as start up the Tomcat the open file count goes up to 6400. Few
>> questions,
>>
>> 1) Why is Solr holding on to all the segments from all the cores - is
>> it because of auto-warmer?
>> 2) How can I reduce the open file count?
>> 3) Is there a way to stop the auto-warmer?
>> 4) Could this be related to "Tomcat returning blank page for every request"?
>>
>> Any ideas?
>>
>> Thanks,
>> -vivek
>>
>> On Fri, Apr 10, 2009 at 1:48 PM, vivek sar wrote:
>> > Hi,
>> >
>> > I was using CommonsHttpSolrServer for indexing, but having two
>> > threads writing (10K batches) at the same time was throwing,
>> >
>> > "ProtocolException: Unbuffered entity enclosing request can not be repeated.
>> "
>> >
>> > I switched to StreamingUpdateSolrServer (using addBeans) and I don't
>> > see the problem anymore. The speed is very fast - getting around
>> > 25k/sec (single thread), but I'm facing another problem. When the
>> > indexer using StreamingUpdateSolrServer is running I'm not able to
>> > send any url request from browser to Solr web app. I just get blank
>> > page. I can't even get to the admin interface. I'm also not able to
>> > shutdown the Tomcat running the Solr webapp when the Indexer is
>> > running. I've to first stop the Indexer app and then stop the Tomcat.
>> > I don't have this problem when using CommonsHttpSolrServer.
>> >
>> > Here is how I'm creating it,
>> >
>> > server = new StreamingUpdateSolrServer(url, 1000,3);
>> >
>> > I simply call server.addBeans(...) on it. Is there anything else I
>> > need to do to make use of StreamingUpdateSolrServer? Why does Tomcat
>> > become unresponsive when Indexer using StreamingUpdateSolrServer is
>> > running (though, indexing happens fine)?
>> >
>> > Thanks,
>> > -vivek
>> >
>
>
Re: Question on StreamingUpdateSolrServer
Posted by Otis Gospodnetic <ot...@yahoo.com>.
One more thing. I don't think this was mentioned, but you can:
- optimize your indices
- use compound index format
That will lower the number of open file handles.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: vivek sar <vi...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, April 10, 2009 5:59:37 PM
> Subject: Re: Question on StreamingUpdateSolrServer
>
> I also noticed that the Solr app has over 6000 file handles open -
>
> "lsof | grep solr | wc -l" - shows 6455
>
> I've 10 cores (using multi-core) managed by the same Solr instance. As
> soon as start up the Tomcat the open file count goes up to 6400. Few
> questions,
>
> 1) Why is Solr holding on to all the segments from all the cores - is
> it because of auto-warmer?
> 2) How can I reduce the open file count?
> 3) Is there a way to stop the auto-warmer?
> 4) Could this be related to "Tomcat returning blank page for every request"?
>
> Any ideas?
>
> Thanks,
> -vivek
>
> On Fri, Apr 10, 2009 at 1:48 PM, vivek sar wrote:
> > Hi,
> >
> > I was using CommonsHttpSolrServer for indexing, but having two
> > threads writing (10K batches) at the same time was throwing,
> >
> > "ProtocolException: Unbuffered entity enclosing request can not be repeated.
> "
> >
> > I switched to StreamingUpdateSolrServer (using addBeans) and I don't
> > see the problem anymore. The speed is very fast - getting around
> > 25k/sec (single thread), but I'm facing another problem. When the
> > indexer using StreamingUpdateSolrServer is running I'm not able to
> > send any url request from browser to Solr web app. I just get blank
> > page. I can't even get to the admin interface. I'm also not able to
> > shutdown the Tomcat running the Solr webapp when the Indexer is
> > running. I've to first stop the Indexer app and then stop the Tomcat.
> > I don't have this problem when using CommonsHttpSolrServer.
> >
> > Here is how I'm creating it,
> >
> > server = new StreamingUpdateSolrServer(url, 1000,3);
> >
> > I simply call server.addBeans(...) on it. Is there anything else I
> > need to do to make use of StreamingUpdateSolrServer? Why does Tomcat
> > become unresponsive when Indexer using StreamingUpdateSolrServer is
> > running (though, indexing happens fine)?
> >
> > Thanks,
> > -vivek
> >
Re: Question on StreamingUpdateSolrServer
Posted by vivek sar <vi...@gmail.com>.
I also noticed that the Solr app has over 6000 file handles open -
"lsof | grep solr | wc -l" - shows 6455
I've 10 cores (using multi-core) managed by the same Solr instance. As
soon as start up the Tomcat the open file count goes up to 6400. Few
questions,
1) Why is Solr holding on to all the segments from all the cores - is
it because of auto-warmer?
2) How can I reduce the open file count?
3) Is there a way to stop the auto-warmer?
4) Could this be related to "Tomcat returning blank page for every request"?
Any ideas?
Thanks,
-vivek
On Fri, Apr 10, 2009 at 1:48 PM, vivek sar <vi...@gmail.com> wrote:
> Hi,
>
> I was using CommonsHttpSolrServer for indexing, but having two
> threads writing (10K batches) at the same time was throwing,
>
> "ProtocolException: Unbuffered entity enclosing request can not be repeated. "
>
> I switched to StreamingUpdateSolrServer (using addBeans) and I don't
> see the problem anymore. The speed is very fast - getting around
> 25k/sec (single thread), but I'm facing another problem. When the
> indexer using StreamingUpdateSolrServer is running I'm not able to
> send any url request from browser to Solr web app. I just get blank
> page. I can't even get to the admin interface. I'm also not able to
> shutdown the Tomcat running the Solr webapp when the Indexer is
> running. I've to first stop the Indexer app and then stop the Tomcat.
> I don't have this problem when using CommonsHttpSolrServer.
>
> Here is how I'm creating it,
>
> server = new StreamingUpdateSolrServer(url, 1000,3);
>
> I simply call server.addBeans(...) on it. Is there anything else I
> need to do to make use of StreamingUpdateSolrServer? Why does Tomcat
> become unresponsive when Indexer using StreamingUpdateSolrServer is
> running (though, indexing happens fine)?
>
> Thanks,
> -vivek
>