You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Supriya Kumar Shyamal <su...@artnology.com> on 2006/10/16 13:40:39 UTC
Parallel Index Search
Hello All,
If I am not mistaken the process of locking the Index by different
objects like IndexReader or Indexwriter, theoratically only one Thread
can access the index at a time.
When we do search on the index it creates a commit lock so the other
thread does not modify the index, so other thread waits until the
previosu therads release the lock, is it right?
So in this case I should say index accessed one by one not parallel?
Its just my speculation, please don't get me wrong.
Because I try to share the same index by 6 instances and since the lock
for 5 instances are disabled and only once instance can modify the
index, at this case I achive the parallel read of the index. Only
disadvantage is that when the index modified then I get
FileNotFoundException, ao I do some kind of respawn the search again.
If I implement the lock mechanism in the DB using the custom locking
then I am afraid the index performance will be reduced but the only
advantage is that I can avoid FNFE.
With Regards,
supriya
--
Mit freundlichen Grüßen / Regards
Supriya Kumar Shyamal
Software Developer
tel +49 (30) 443 50 99 -22
fax +49 (30) 443 50 99 -99
email supriya.shyamal@artnology.com
___________________________
artnology GmbH
Milastr. 4
10437 Berlin
___________________________
http://www.artnology.com
__________________________________________________________________________
News / Aktuelle Projekte:
* artnology gewinnt Ausschreibung des Bundesministeriums des Innern:
Softwarelösung für die Verwaltung der Sammlung zeitgenössischer
Kunstwerke zur kulturellen Repräsentation des Bundes.
Projektreferenzen:
* Globaler eShop und Corporate-Site für Springer: www.springeronline.com
* E-Detailing-Portal für Novartis: www.interaktiv.novartis.de
* Service-Center-Plattform für Biogen: www.ms-life.de
* eCRM-System für Grünenthal: www.gruenenthal.com
___________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Parallel Index Search
Posted by Michael McCandless <lu...@mikemccandless.com>.
Supriya Kumar Shyamal wrote:
>>> If I am not mistaken the process of locking the Index by different
>>> objects like IndexReader or Indexwriter, theoratically only one
>>> Thread can access the index at a time.
>>
>> Actually, only one writer can write to the index at once. Multiple
>> readers can read from the index.
>>
>> On top of that, multiple threads may share one writer and one reader.
>> Ie, these classes are thread safe.
>>
>>> When we do search on the index it creates a commit lock so the other
>>> thread does not modify the index, so other thread waits until the
>>> previosu therads release the lock, is it right?
>>>
>>> So in this case I should say index accessed one by one not parallel?
>>
>> The commit lock is only held while a reader is loading the index and
>> while a writer is "committing" its changes to the index. These times
>> should be brief. Whereas, the write lock is held for the entire time
>> that a writer is open.
> But IndexSearcher open index using INdexReader right?
Yes, IndexSearcher will open an IndexReader. So every time you open
it, the commit lock is acquired, the IndexReader reads all segments
details from the index, and then the commit lock is released. But,
this lock needs to be shared across all your machines, so that the
writer is prevented from committing while a reader is reading the
index.
Aside: note that you should not be re-opening an IndexSearcher with
every search. It's much more efficient to open it (and reopen it
every so often) and then share that one instance across many searches.
>>
>>> Its just my speculation, please don't get me wrong.
>>>
>>> Because I try to share the same index by 6 instances and since the
>>> lock for 5 instances are disabled and only once instance can modify
>>> the index, at this case I achive the parallel read of the index. Only
>>> disadvantage is that when the index modified then I get
>>> FileNotFoundException, ao I do some kind of respawn the search again.
>>
>> You should not have to disable locking to do this sharing; in fact,
>> disabling locking will lead to the "FileNotFoundException" on
>> instantiating a reader when a writer is committing.
>>
>>> If I implement the lock mechanism in the DB using the custom locking
>>> then I am afraid the index performance will be reduced but the only
>>> advantage is that I can avoid FNFE.
>>
>> There are also open issues at least over NFS eg:
>>
>> http://issues.apache.org/jira/browse/LUCENE-673
>>
>> Whereby even with proper (native) locking it's still possible to
>> hit exceptions due to caching in the NFS client. Are you using NFS?
>>
> Yes I am using NFS and for sharing I use the READ-ONLY FS mount so that
> the search instance cannot do any modification by mistake , otherwise it
> can corrupt the index.
OK, Lucene currently hits intermittant FileNotFoundException over NFS
(see that issue I listed for the gory details). Furthermore (worse),
correct locking (native locking) does not in fact fix the issue.
We are working on small changes to how Lucene stores the index in the
filesystem ("lockless commits") that do away with the commit lock
entirely. This not only enables readers to actually be entirely
read-only, but it also allows Lucene to work properly over NFS.
Unfortunately it's going to be a while before this is actually
available (it's still being developed). In the mean time, one
workaround is to do something similar to the Solr project (built on
top of Lucene) which takes snapshots of the index at "known safe"
times and then readers read from these snapshots instead of the live
index.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Parallel Index Search
Posted by Supriya Kumar Shyamal <su...@artnology.com>.
Michael McCandless wrote:
> Supriya Kumar Shyamal wrote:
>
>> If I am not mistaken the process of locking the Index by different
>> objects like IndexReader or Indexwriter, theoratically only one
>> Thread can access the index at a time.
>
> Actually, only one writer can write to the index at once. Multiple
> readers can read from the index.
>
> On top of that, multiple threads may share one writer and one reader.
> Ie, these classes are thread safe.
>
>> When we do search on the index it creates a commit lock so the other
>> thread does not modify the index, so other thread waits until the
>> previosu therads release the lock, is it right?
>>
>> So in this case I should say index accessed one by one not parallel?
>
> The commit lock is only held while a reader is loading the index and
> while a writer is "committing" its changes to the index. These times
> should be brief. Whereas, the write lock is held for the entire time
> that a writer is open.
But IndexSearcher open index using INdexReader right?
>
>> Its just my speculation, please don't get me wrong.
>>
>> Because I try to share the same index by 6 instances and since the
>> lock for 5 instances are disabled and only once instance can modify
>> the index, at this case I achive the parallel read of the index. Only
>> disadvantage is that when the index modified then I get
>> FileNotFoundException, ao I do some kind of respawn the search again.
>
> You should not have to disable locking to do this sharing; in fact,
> disabling locking will lead to the "FileNotFoundException" on
> instantiating a reader when a writer is committing.
>
>> If I implement the lock mechanism in the DB using the custom locking
>> then I am afraid the index performance will be reduced but the only
>> advantage is that I can avoid FNFE.
>
> There are also open issues at least over NFS eg:
>
> http://issues.apache.org/jira/browse/LUCENE-673
>
> Whereby even with proper (native) locking it's still possible to
> hit exceptions due to caching in the NFS client. Are you using NFS?
>
Yes I am using NFS and for sharing I use the READ-ONLY FS mount so that
the search instance cannot do any modification by mistake , otherwise it
can corrupt the index.
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
Mit freundlichen Grüßen / Regards
Supriya Kumar Shyamal
Software Developer
tel +49 (30) 443 50 99 -22
fax +49 (30) 443 50 99 -99
email supriya.shyamal@artnology.com
___________________________
artnology GmbH
Milastr. 4
10437 Berlin
___________________________
http://www.artnology.com
__________________________________________________________________________
News / Aktuelle Projekte:
* artnology gewinnt Ausschreibung des Bundesministeriums des Innern:
Softwarelösung für die Verwaltung der Sammlung zeitgenössischer
Kunstwerke zur kulturellen Repräsentation des Bundes.
Projektreferenzen:
* Globaler eShop und Corporate-Site für Springer: www.springeronline.com
* E-Detailing-Portal für Novartis: www.interaktiv.novartis.de
* Service-Center-Plattform für Biogen: www.ms-life.de
* eCRM-System für Grünenthal: www.gruenenthal.com
___________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Parallel Index Search
Posted by Michael McCandless <lu...@mikemccandless.com>.
Supriya Kumar Shyamal wrote:
> If I am not mistaken the process of locking the Index by different
> objects like IndexReader or Indexwriter, theoratically only one Thread
> can access the index at a time.
Actually, only one writer can write to the index at once. Multiple
readers can read from the index.
On top of that, multiple threads may share one writer and one reader.
Ie, these classes are thread safe.
> When we do search on the index it creates a commit lock so the other
> thread does not modify the index, so other thread waits until the
> previosu therads release the lock, is it right?
>
> So in this case I should say index accessed one by one not parallel?
The commit lock is only held while a reader is loading the index and
while a writer is "committing" its changes to the index. These times
should be brief. Whereas, the write lock is held for the entire time
that a writer is open.
> Its just my speculation, please don't get me wrong.
>
> Because I try to share the same index by 6 instances and since the lock
> for 5 instances are disabled and only once instance can modify the
> index, at this case I achive the parallel read of the index. Only
> disadvantage is that when the index modified then I get
> FileNotFoundException, ao I do some kind of respawn the search again.
You should not have to disable locking to do this sharing; in fact,
disabling locking will lead to the "FileNotFoundException" on
instantiating a reader when a writer is committing.
> If I implement the lock mechanism in the DB using the custom locking
> then I am afraid the index performance will be reduced but the only
> advantage is that I can avoid FNFE.
There are also open issues at least over NFS eg:
http://issues.apache.org/jira/browse/LUCENE-673
Whereby even with proper (native) locking it's still possible to
hit exceptions due to caching in the NFS client. Are you using NFS?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org