You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Bernhard Messer <Be...@intrafind.de> on 2004/08/15 16:47:15 UTC
optimize TermVectorsReader, remove synchronization from code
hi all,
as promised last week, i would like to contribute a patch which allows
to remove all synchronization from TermVectorsReader class. This patch
dramatically improves the overall performance using one IndexReader
within several threads. To show the difference, there is a new JUnit
test which shows the difference.
calculated time without patch:
threadcount: 1 average term vector time: 385
threadcount: 2 average term vector time: 1013
threadcount: 4 average term vector time: 2879
threadcount: 6 average term vector time: 11311
threadcount: 8 average term vector time: 29323
threadcount: 10 average term vector time: 35797
time result using the patch:
threadcount: 1 average term vector time: 379
threadcount: 2 average term vector time: 736
threadcount: 4 average term vector time: 877
threadcount: 6 average term vector time: 947
threadcount: 8 average term vector time: 2407
threadcount: 10 average term vector time: 2648
If lucene developers think this patch is useful and might has a chance
to become committed, i can create a new Bugzilla issue and add the files
there.
best regards
Bernhard
Re: optimize TermVectorsReader, remove synchronization from code
Posted by Bernhard Messer <Be...@intrafind.de>.
Otis,
i will check everything tonight and add a new patch to Bugzilla.
regards
Bernhard
Otis Gospodnetic wrote:
>I was going to consider applying this, but the included diff has
>wrapped lines. Is there a Bugzilla entry with this code attached?
>
>Also, is there only SegmentReader.java diff here, or am I missing a
>piece of diff? I don't see any synchronized methods/blocks removed
>from the code, so I'm confused about this optimization.
>
>Otis
>
>--- Bernhard Messer <Be...@intrafind.de> wrote:
>
>
>
>>Sorry, but there was a bug in the patch i provided several minutes
>>ago.
>>A NullpointerException can occur in SegmentReader doClose method. The
>>
>>change in the new diff file checks if the ThreadLocal object was
>>created
>>and is not null before trying to get the TermVectorReader from it.
>>
>>best regards
>>Bernhard
>>
>>
>>
>>>Index: SegmentReader.java
>>>
>>>
>>===================================================================
>>RCS file:
>>
>>
>>
>/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/SegmentReader.java,v
>
>
>>retrieving revision 1.25
>>diff -u -r1.25 SegmentReader.java
>>--- SegmentReader.java 11 Aug 2004 17:37:52 -0000 1.25
>>+++ SegmentReader.java 15 Aug 2004 15:07:47 -0000
>>@@ -42,8 +42,7 @@
>> private FieldsReader fieldsReader;
>>
>> TermInfosReader tis;
>>- TermVectorsReader termVectorsReader;
>>-
>>+
>> BitVector deletedDocs = null;
>> private boolean deletedDocsDirty = false;
>> private boolean normsDirty = false;
>>@@ -51,6 +50,8 @@
>>
>> InputStream freqStream;
>> InputStream proxStream;
>>+
>>+ private ThreadLocal termVectorsLocal = null;
>>
>> // Compound File Reader when based on a compound file segment
>> CompoundFileReader cfsReader = null;
>>@@ -128,7 +129,17 @@
>> openNorms(cfsDir);
>>
>> if (fieldInfos.hasVectors()) { // open term vector files only as
>>needed
>>- termVectorsReader = new TermVectorsReader(cfsDir, segment,
>>fieldInfos);
>>+ final Directory dir = cfsDir;
>>+ termVectorsLocal = new ThreadLocal() {
>>+ protected synchronized Object initialValue() {
>>+ try {
>>+ return new TermVectorsReader(dir, segment, fieldInfos);
>>+ } catch (IOException ioe) {
>>+ ioe.printStackTrace();
>>+ return null;
>>+ }
>>+ }
>>+ };
>> }
>> }
>>
>>@@ -164,8 +175,13 @@
>> proxStream.close();
>>
>> closeNorms();
>>- if (termVectorsReader != null) termVectorsReader.close();
>>-
>>+ if (termVectorsLocal != null) {
>>+ TermVectorsReader termVectorsReader =
>>(TermVectorsReader)termVectorsLocal.get();
>>+ if (termVectorsReader != null) {
>>+ termVectorsReader.close();
>>+ }
>>+ }
>>+
>> if (cfsReader != null)
>> cfsReader.close();
>> }
>>@@ -408,6 +424,15 @@
>> FieldInfo fi = fieldInfos.fieldInfo(field);
>> if (fi == null || !fi.storeTermVector) return null;
>>
>>+ if (termVectorsLocal == null) {
>>+ return null;
>>+ }
>>+
>>+ TermVectorsReader termVectorsReader =
>>(TermVectorsReader)termVectorsLocal.get();
>>+ if (termVectorsReader == null) {
>>+ return null;
>>+ }
>>+
>> return termVectorsReader.get(docNumber, field);
>> }
>>
>>@@ -419,9 +444,14 @@
>> * If no such fields existed, the method returns null.
>> */
>> public TermFreqVector[] getTermFreqVectors(int docNumber) {
>>- if (termVectorsReader == null)
>>- return null;
>>-
>>+ if (termVectorsLocal == null) {
>>+ return null;
>>+ }
>>+
>>+ TermVectorsReader termVectorsReader =
>>(TermVectorsReader)termVectorsLocal.get();
>>+ if (termVectorsReader == null) {
>>+ return null;
>>+ }
>> return termVectorsReader.get(docNumber);
>> }
>> }
>>
>>
>>
>---------------------------------------------------------------------
>
>
>>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
>
Re: optimize TermVectorsReader, remove synchronization from code
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I was going to consider applying this, but the included diff has
wrapped lines. Is there a Bugzilla entry with this code attached?
Also, is there only SegmentReader.java diff here, or am I missing a
piece of diff? I don't see any synchronized methods/blocks removed
from the code, so I'm confused about this optimization.
Otis
--- Bernhard Messer <Be...@intrafind.de> wrote:
> Sorry, but there was a bug in the patch i provided several minutes
> ago.
> A NullpointerException can occur in SegmentReader doClose method. The
>
> change in the new diff file checks if the ThreadLocal object was
> created
> and is not null before trying to get the TermVectorReader from it.
>
> best regards
> Bernhard
>
> > Index: SegmentReader.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/SegmentReader.java,v
> retrieving revision 1.25
> diff -u -r1.25 SegmentReader.java
> --- SegmentReader.java 11 Aug 2004 17:37:52 -0000 1.25
> +++ SegmentReader.java 15 Aug 2004 15:07:47 -0000
> @@ -42,8 +42,7 @@
> private FieldsReader fieldsReader;
>
> TermInfosReader tis;
> - TermVectorsReader termVectorsReader;
> -
> +
> BitVector deletedDocs = null;
> private boolean deletedDocsDirty = false;
> private boolean normsDirty = false;
> @@ -51,6 +50,8 @@
>
> InputStream freqStream;
> InputStream proxStream;
> +
> + private ThreadLocal termVectorsLocal = null;
>
> // Compound File Reader when based on a compound file segment
> CompoundFileReader cfsReader = null;
> @@ -128,7 +129,17 @@
> openNorms(cfsDir);
>
> if (fieldInfos.hasVectors()) { // open term vector files only as
> needed
> - termVectorsReader = new TermVectorsReader(cfsDir, segment,
> fieldInfos);
> + final Directory dir = cfsDir;
> + termVectorsLocal = new ThreadLocal() {
> + protected synchronized Object initialValue() {
> + try {
> + return new TermVectorsReader(dir, segment, fieldInfos);
> + } catch (IOException ioe) {
> + ioe.printStackTrace();
> + return null;
> + }
> + }
> + };
> }
> }
>
> @@ -164,8 +175,13 @@
> proxStream.close();
>
> closeNorms();
> - if (termVectorsReader != null) termVectorsReader.close();
> -
> + if (termVectorsLocal != null) {
> + TermVectorsReader termVectorsReader =
> (TermVectorsReader)termVectorsLocal.get();
> + if (termVectorsReader != null) {
> + termVectorsReader.close();
> + }
> + }
> +
> if (cfsReader != null)
> cfsReader.close();
> }
> @@ -408,6 +424,15 @@
> FieldInfo fi = fieldInfos.fieldInfo(field);
> if (fi == null || !fi.storeTermVector) return null;
>
> + if (termVectorsLocal == null) {
> + return null;
> + }
> +
> + TermVectorsReader termVectorsReader =
> (TermVectorsReader)termVectorsLocal.get();
> + if (termVectorsReader == null) {
> + return null;
> + }
> +
> return termVectorsReader.get(docNumber, field);
> }
>
> @@ -419,9 +444,14 @@
> * If no such fields existed, the method returns null.
> */
> public TermFreqVector[] getTermFreqVectors(int docNumber) {
> - if (termVectorsReader == null)
> - return null;
> -
> + if (termVectorsLocal == null) {
> + return null;
> + }
> +
> + TermVectorsReader termVectorsReader =
> (TermVectorsReader)termVectorsLocal.get();
> + if (termVectorsReader == null) {
> + return null;
> + }
> return termVectorsReader.get(docNumber);
> }
> }
>
> >
---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: optimize TermVectorsReader, remove synchronization from code
Posted by Bernhard Messer <Be...@intrafind.de>.
Sorry, but there was a bug in the patch i provided several minutes ago.
A NullpointerException can occur in SegmentReader doClose method. The
change in the new diff file checks if the ThreadLocal object was created
and is not null before trying to get the TermVectorReader from it.
best regards
Bernhard