You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Adrian Sutton <ad...@symphonious.net> on 2007/09/10 07:33:06 UTC
DirectSolrConnection, write.lock and Too Many Open Files
Hi all,
We've been using DirectSolrConnection in an application with low
overall load, but proportionally a lot of updates (nearly as many
updates as searches). While I love the fact that I don't have to deal
with HTTP, we're seeing a couple of problems.
After a while we start getting exceptions thrown because of a timeout
in acquiring write.lock. It's quite possible that this occurs
whenever two updates are attempted at the same time - is
DirectSolrConnection intended to be thread safe?
The other problem is that after some time we get a "Too Many Open
Files" error when autocommit fires. Usually this happens once or
twice and then stops and things rock along as usual. It's quite
possible this is actually us leaking file handles from elsewhere or
it could be because of the way we're using DirectSolrConnection or it
just could be we need to optimize more. Any hints on how to decide
which of those it is?
At the moment, we create a single DirectSolrConnection instance and
use it for all searches and updates. Previously we've created a new
DirectSolrConnection for each thread and the same problems occured.
The entire tomcat instance is automatically restarted every 12 hours
to deploy updates to the application.
I could of course switch to using the Solr webapp since we're running
in Tomcat anyway, however I really like the ability to have a single
WAR file that contains everything and also not have to worry about
actually making HTTP requests and the complexity that adds.
Any hints would be greatly appreciated, it looks like
DirectSolrConnection isn't heavily used but it is a really cool idea
that I'd suggest is worth pursuing.
Thanks in advance,
Adrian Sutton
http://www.symphonious.net
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Mike Klaas <mi...@gmail.com>.
On 10-Sep-07, at 12:32 PM, Brian Whitman wrote:
>
> On Sep 10, 2007, at 1:56 PM, Mike Klaas wrote:
>>
>> I'm a little worried about Solr getting a bad rap for being buggy
>> or not well-documented due to people trying to use it embeddedly.
>> It seems that we need to do one of two things:
>> 1. Someone lead the effort to clean up and organize the
>> situation, develop and test the relevant api, and document it to
>> the standards of the rest of Solr.
>> 2. Remove references to embedding Solr (including out-of-date
>> wiki pages) if we do not have a maintainer for this functionality.
>>
>
> We have a desktop app in alpha now that uses Solr as its data store
> backend via DirectSolrConnection. It makes a lot of sense in our
> app especially as the web-app side of the app is also built on
> Solr. We have a lot of people banging on the code and other than
> the thread lock issue Adrian and I mentioned, there's been
> absolutely no problem -- it's fast, stable, relatively low
> overhead, x-platform and does exactly what we need.
Have you tried setting the write lock to "simple" (solr trunk)?
> I would volunteer to be the DSC maintainer as I probably use it the
> most, but unfortunately I am not a competent java dev (Ryan can
> attest to this.) I will volunteer to clean up the end user docs,
> specifically I owe the wiki a non-objc version of the JNI code.
Thanks!
-Mike
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Brian Whitman <br...@variogr.am>.
On Sep 10, 2007, at 1:56 PM, Mike Klaas wrote:
>
> I'm a little worried about Solr getting a bad rap for being buggy
> or not well-documented due to people trying to use it embeddedly.
> It seems that we need to do one of two things:
> 1. Someone lead the effort to clean up and organize the situation,
> develop and test the relevant api, and document it to the standards
> of the rest of Solr.
> 2. Remove references to embedding Solr (including out-of-date wiki
> pages) if we do not have a maintainer for this functionality.
>
We have a desktop app in alpha now that uses Solr as its data store
backend via DirectSolrConnection. It makes a lot of sense in our app
especially as the web-app side of the app is also built on Solr. We
have a lot of people banging on the code and other than the thread
lock issue Adrian and I mentioned, there's been absolutely no problem
-- it's fast, stable, relatively low overhead, x-platform and does
exactly what we need.
I would volunteer to be the DSC maintainer as I probably use it the
most, but unfortunately I am not a competent java dev (Ryan can
attest to this.) I will volunteer to clean up the end user docs,
specifically I owe the wiki a non-objc version of the JNI code.
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Mike Klaas <mi...@gmail.com>.
On 10-Sep-07, at 4:42 AM, Brian Whitman wrote:
>
> On Sep 10, 2007, at 1:33 AM, Adrian Sutton wrote:
>> After a while we start getting exceptions thrown because of a
>> timeout in acquiring write.lock. It's quite possible that this
>> occurs whenever two updates are attempted at the same time - is
>> DirectSolrConnection intended to be thread safe?
>>
>
> We use DirectSolrConnection via JNI in a couple of client apps that
> sometimes have 100s of thousands of new docs as fast as Solr will
> have them. It would crash relentlessly if I didn't force all calls
> to update or query to be on the same thread using objc's
> @synchronized and a message queue. I never narrowed down if this
> was a solr issue or a JNI one.
Sounds like we should put a disclaimer somewhere--I don't get the
impression that DirectSolrConnection is very polished, tested, or
documented.
The problem largely stems from Solr not being designed to be used as
search library code. I don't think any core devs use it in this mode
(except perhaps Ryan?), and so it hasn't got a lot of love.
Embedded Solr/DSC seem to be popular despite being not well-
publicized; eg.:
http://www.symphonious.net/category/apache/
I'm a little worried about Solr getting a bad rap for being buggy or
not well-documented due to people trying to use it embeddedly. It
seems that we need to do one of two things:
1. Someone lead the effort to clean up and organize the situation,
develop and test the relevant api, and document it to the standards
of the rest of Solr.
2. Remove references to embedding Solr (including out-of-date wiki
pages) if we do not have a maintainer for this functionality.
ciao,
-Mike
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Yonik Seeley <yo...@apache.org>.
On 9/10/07, Adrian Sutton <ad...@symphonious.net> wrote:
> Can Solr as a web app handle multiple updates at
> once or does it synchronize to avoid it?
Yep... things aren't synchronized at the top level and are designed to
be thread-safe.
-Yonik
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Brian Whitman <br...@variogr.am>.
On Sep 10, 2007, at 5:00 PM, Mike Klaas wrote:
> On 10-Sep-07, at 1:50 PM, Adrian Sutton wrote:
>
>>> We use DirectSolrConnection via JNI in a couple of client apps
>>> that sometimes have 100s of thousands of new docs as fast as Solr
>>> will have them. It would crash relentlessly if I didn't force all
>>> calls to update or query to be on the same thread using objc's
>>> @synchronized and a message queue. I never narrowed down if this
>>> was a solr issue or a JNI one.
>>
>> That doesn't sound promising. I'll throw in synchronization around
>> the update code and see what happens. That's doesn't seem good for
>> performance though. Can Solr as a web app handle multiple updates
>> at once or does it synchronize to avoid it?
>
> Solr can handle multiple simultaneous updates. The entire request
> processing is concurrent, as is the document analysis. Only the
> final write is synchronized (this includes lucene segment merging).
>
Yes, i do want to disclaim that it's very likely my thread problems
are an implementation detail w/ JNI, nothing to do w/ DSC.
-b
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Mike Klaas <mi...@gmail.com>.
On 10-Sep-07, at 1:50 PM, Adrian Sutton wrote:
>> We use DirectSolrConnection via JNI in a couple of client apps
>> that sometimes have 100s of thousands of new docs as fast as Solr
>> will have them. It would crash relentlessly if I didn't force all
>> calls to update or query to be on the same thread using objc's
>> @synchronized and a message queue. I never narrowed down if this
>> was a solr issue or a JNI one.
>
> That doesn't sound promising. I'll throw in synchronization around
> the update code and see what happens. That's doesn't seem good for
> performance though. Can Solr as a web app handle multiple updates
> at once or does it synchronize to avoid it?
Solr can handle multiple simultaneous updates. The entire request
processing is concurrent, as is the document analysis. Only the
final write is synchronized (this includes lucene segment merging).
In the future, segment merging will occur in a separate thread,
further improving concurrency.
-Mike
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Adrian Sutton <ad...@symphonious.net>.
> We use DirectSolrConnection via JNI in a couple of client apps that
> sometimes have 100s of thousands of new docs as fast as Solr will
> have them. It would crash relentlessly if I didn't force all calls
> to update or query to be on the same thread using objc's
> @synchronized and a message queue. I never narrowed down if this
> was a solr issue or a JNI one.
That doesn't sound promising. I'll throw in synchronization around
the update code and see what happens. That's doesn't seem good for
performance though. Can Solr as a web app handle multiple updates at
once or does it synchronize to avoid it?
Thanks,
Adrian Sutton
http://www.symphonious.net
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Brian Whitman <br...@variogr.am>.
On Sep 10, 2007, at 1:33 AM, Adrian Sutton wrote:
> After a while we start getting exceptions thrown because of a
> timeout in acquiring write.lock. It's quite possible that this
> occurs whenever two updates are attempted at the same time - is
> DirectSolrConnection intended to be thread safe?
>
We use DirectSolrConnection via JNI in a couple of client apps that
sometimes have 100s of thousands of new docs as fast as Solr will
have them. It would crash relentlessly if I didn't force all calls to
update or query to be on the same thread using objc's @synchronized
and a message queue. I never narrowed down if this was a solr issue
or a JNI one.
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Adrian Sutton <ad...@symphonious.net>.
On 11/09/2007, at 9:48 AM, Ryan McKinley wrote:
> try: ulimit -n
>
> ulimit on its own is something else. On my machine I get:
>
> ryan@box:~$ ulimit
> unlimited
> ryan@box:~$ cat /proc/sys/fs/file-max
> 364770
> ryan@box:~$ ulimit -n
> 1024
>
>
> I have to run:
> ulimit -n 20000
>
> to get lucene to run w/ a large index...
Bingo, I'm an idiot - or rather, I now know *why* I'm an idiot. :)
I'll give it a go.
Also, this is likely to be the cause of my write.lock problems - the
Too many files exception just occured and the write.lock file gets
left around (should have seen that one coming too).
Thanks for your help, I'm anticipating that this will solve our
problems.
Regards,
Adrian Sutton
http://www.symphonious.net
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Ryan McKinley <ry...@gmail.com>.
>
> I've done a bit of poking on the server and ulimit doesn't seem to be
> the problem:
> e2wiki:~$ ulimit
> unlimited
> e2wiki:~$ cat /proc/sys/fs/file-max
> 170355
try: ulimit -n
ulimit on its own is something else. On my machine I get:
ryan@box:~$ ulimit
unlimited
ryan@box:~$ cat /proc/sys/fs/file-max
364770
ryan@box:~$ ulimit -n
1024
I have to run:
ulimit -n 20000
to get lucene to run w/ a large index...
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Adrian Sutton <ad...@symphonious.net>.
On 11/09/2007, at 8:46 AM, Ryan McKinley wrote:
> lucene opens a lot of files. It can easily get beyond 1024. (I
> think the default). I'm no expert on how the file handling works,
> but I think more files are open if you are searching and writing at
> the same time.
>
> If you can't increase the limit you can try:
> <useCompoundFile>true</useCompoundFile>
>
> It is slower, but if you are unable to change the ulimit on the
> deployed machines....
I've done a bit of poking on the server and ulimit doesn't seem to be
the problem:
e2wiki:~$ ulimit
unlimited
e2wiki:~$ cat /proc/sys/fs/file-max
170355
So there's either something going on behind my back (quite possible,
it's a VM) or lucene is opening a really insane number of files. I
did check that those values were the same for the tomcat55 user that
Tomcat actually runs as. An lsof -p on the Tomcat process always
shows 40 files in use, the total open files sits around 1000-1500
even when reindexing all the content. I'll watch it a bit more over
time and see what happens.
I notice that Confluence recommends at least 200000 for the max file
limit, at least before they switched to compound indexing so it's
possible that the 170355 limit could be reached, but it seems
unlikely with our load.
> If you need to use this in production soon, I'd suggest sticking
> with 1.2 for a while. There has been a LOT of action in trunk and
> it may be good to let it settle before upgrading a production system.
>
> You should not need to upgrade to fix the write.lock and Too Many
> Open Files problem. Try increasing ulimit or using a compoundfile
> before upgrading.
We're quite a way off of real production, it's just internal use at
the moment (on the real product server, but we're a small company so
we can handle having some problems). I'll try out the current nightly
build and see how it goes, as much as anything out of interest but
probably won't pull new builds very often.
Thanks again,
Adrian Sutton
http://www.symphonious.net
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Ryan McKinley <ry...@gmail.com>.
Adrian Sutton wrote:
> On 11/09/2007, at 7:21 AM, Ryan McKinley wrote:
>
>>> The other problem is that after some time we get a "Too Many Open
>>> Files" error when autocommit fires.
>>
>> Have you checked your ulimit settings?
>>
>> http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
>>
>>
>> ulimit -n <number>.
>
> Yeah I'm aware of the ulimit, I'm just keen to identify what's causing
> it to happen before starting to increase limits. Given the write.lock
> errors as well I'm particularly suspicious of it. That said, most likely
> it happens whenever a search and a write are happening at the same time
> and two sets of the files get opened which is enough to kick it over the
> limit. The fact that it fixes itself is a good indication that it's not
> a file handle leak.
lucene opens a lot of files. It can easily get beyond 1024. (I think
the default). I'm no expert on how the file handling works, but I think
more files are open if you are searching and writing at the same time.
If you can't increase the limit you can try:
<useCompoundFile>true</useCompoundFile>
It is slower, but if you are unable to change the ulimit on the deployed
machines....
>
>> As mike mentioned, you may also want to use 'single' as the lockType.
>> In solrconfig set:
>>
>> <indexDefaults>
>> ...
>> <lockType>single</lockType>
>> </indexDefaults>
>
> I'll give that a go. Looks like it didn't make it into Solr 1.2 so I'll
> try upgrading to the nightly build.
>
If you need to use this in production soon, I'd suggest sticking with
1.2 for a while. There has been a LOT of action in trunk and it may be
good to let it settle before upgrading a production system.
You should not need to upgrade to fix the write.lock and Too Many Open
Files problem. Try increasing ulimit or using a compoundfile before
upgrading.
>
> Just when you think you know everything on the wiki....
someone finally updates it!
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Adrian Sutton <ad...@symphonious.net>.
On 11/09/2007, at 7:21 AM, Ryan McKinley wrote:
>> The other problem is that after some time we get a "Too Many Open
>> Files" error when autocommit fires.
>
> Have you checked your ulimit settings?
>
> http://wiki.apache.org/lucene-java/
> LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
>
> ulimit -n <number>.
Yeah I'm aware of the ulimit, I'm just keen to identify what's
causing it to happen before starting to increase limits. Given the
write.lock errors as well I'm particularly suspicious of it. That
said, most likely it happens whenever a search and a write are
happening at the same time and two sets of the files get opened which
is enough to kick it over the limit. The fact that it fixes itself is
a good indication that it's not a file handle leak.
> As mike mentioned, you may also want to use 'single' as the
> lockType. In solrconfig set:
>
> <indexDefaults>
> ...
> <lockType>single</lockType>
> </indexDefaults>
I'll give that a go. Looks like it didn't make it into Solr 1.2 so
I'll try upgrading to the nightly build.
>> I could of course switch to using the Solr webapp since we're
>> running in Tomcat anyway, however I really like the ability to
>> have a single WAR file that contains everything and also not have
>> to worry about actually making HTTP requests and the complexity
>> that adds.
>
> This sounds like a good candidate to try solrj:
> http://wiki.apache.org/solr/Solrj
>
> This way you write your app independent of how you connect to
> solr. It also takes care of the XML parsing for you and lets you
> work with objects rather then strings.
Just when you think you know everything on the wiki.... This looks
fantastic I wish I knew about it a couple of months ago. Not sure if
I can resolve the JVM 1.4 vs 1.5 issue for that API though (we have
multiple deploy targets, it's painful but needed).
> ryan
Lots of stuff to play with.
Thanks,
Adrian Sutton
http://www.symphonious.net
Re: DirectSolrConnection, write.lock and Too Many Open Files
Posted by Ryan McKinley <ry...@gmail.com>.
>
> The other problem is that after some time we get a "Too Many Open Files"
> error when autocommit fires.
Have you checked your ulimit settings?
http://wiki.apache.org/lucene-java/LuceneFAQ#head-48921635adf2c968f7936dc07d51dfb40d638b82
ulimit -n <number>.
As mike mentioned, you may also want to use 'single' as the lockType.
In solrconfig set:
<indexDefaults>
...
<lockType>single</lockType>
</indexDefaults>
>
> I could of course switch to using the Solr webapp since we're running in
> Tomcat anyway, however I really like the ability to have a single WAR
> file that contains everything and also not have to worry about actually
> making HTTP requests and the complexity that adds.
>
This sounds like a good candidate to try solrj:
http://wiki.apache.org/solr/Solrj
This way you write your app independent of how you connect to solr. It
also takes care of the XML parsing for you and lets you work with
objects rather then strings.
ryan