You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2008/12/09 21:03:44 UTC

[jira] Created: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Use OpenBitSet instead of BitVector in SegmentReader
----------------------------------------------------

                 Key: LUCENE-1485
                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
    Affects Versions: 2.4
            Reporter: Jason Rutherglen
            Priority: Minor


Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator for in SegmentTermDocs will speed things up more.

bit set size: 10,485,760
set bits count: 524,032
openbitset: 68
bitvector: 89

24% speed increase.

I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655862#action_12655862 ] 

Yonik Seeley commented on LUCENE-1485:
--------------------------------------

bq.  It is somewhat surprising given OpenBitSet is supposed to be the "fastest" bitset

For doing population counts, and intersection/union population counts, yes.  And it's also "Open" so if there is a faster method of doing something, it can still be done.  The point was not to make a faster get(bitnum) - that's really going to be in the noise and certainly not worth writing a whole new BitSet class over.

w.r.t. Lucene's BitVector, it's byte based, so is likely to be slightly faster on x86, esp 32 bit.  No need to check a long when you can check a byte for a flipped bit.

bq. It seems that Lucene should have ways to incorporate new bitset implementations in the future using interfaces and things.

DocIdSet?

> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: TestDeletedDocsSpeed.java
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655385#action_12655385 ] 

Grant Ingersoll commented on LUCENE-1485:
-----------------------------------------

Is the "slightly" in the noise?

> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: TestDeletedDocsSpeed.java
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654994#action_12654994 ] 

Jason Rutherglen commented on LUCENE-1485:
------------------------------------------

This above test was using the -client option in the JVM on Mac OS X.  Using -server the numbers look almost the same for OpenBitSet and BitVector with BitVector being slightly faster.  

> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: TestDeletedDocsSpeed.java
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1485:
-------------------------------------

    Description: 
Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.

bit set size: 10,485,760
set bits count: 524,032
openbitset: 68
bitvector: 89

24% speed increase.

I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

  was:
Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator for in SegmentTermDocs will speed things up more.

bit set size: 10,485,760
set bits count: 524,032
openbitset: 68
bitvector: 89

24% speed increase.

I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  


> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1485:
-------------------------------------

    Attachment: TestDeletedDocsSpeed.java

TestDeletedDocsSpeed.java

Executes get on BitVector and OpenBitSet.  FastGet is called on OpenBitSet.  

> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: TestDeletedDocsSpeed.java
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Commented: (LUCENE-1485) Use OpenBitSet instead of BitVector in SegmentReader

Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655849#action_12655849 ] 

Jason Rutherglen commented on LUCENE-1485:
------------------------------------------

Grant write: "Is the "slightly" in the noise? "

Seems to be.  Perhaps it needs more performance tests.  It is somewhat surprising given OpenBitSet is supposed to be the "fastest" bitset.  It seems that Lucene should have ways to incorporate new bitset implementations in the future using interfaces and things.  That being said it would be great if in Lucene 3.0 the entire IndexReader class tree was rewritten to not be such as mess with the locking, reopen, and ref counting.  Marvin is proposing some good ideas to make it all more pluggable.  I need to spend some time with folks figuring what APIs would be optimal for not tying all the APIs together like the twisted mess it is now.  For example, IndexReader shouldn't have a static open method attached to it.  It seems like new index features like column stride fields implemented in todays system would exacerbate the problem because then there's more code that is impossible to customize if desired.  SegmentMerger needs to be pluggable as today it cannot be customized without possibly breaking the entirety of Lucene, and the custom code cannot be checked in as a contrib.  There more to write but I should save it for a more structured and timely discussion.

> Use OpenBitSet instead of BitVector in SegmentReader
> ----------------------------------------------------
>
>                 Key: LUCENE-1485
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1485
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Index
>    Affects Versions: 2.4
>            Reporter: Jason Rutherglen
>            Priority: Minor
>         Attachments: TestDeletedDocsSpeed.java
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Tried out BitVector.get vs OpenBitSet.get here's the results which are about the same after running 25 times in milliseconds.  It is assumed that implementing DocIdSetIterator in SegmentTermDocs will speed things up more.
> bit set size: 10,485,760
> set bits count: 524,032
> openbitset: 68
> bitvector: 89
> 24% speed increase.
> I will implement a patch that adds the WriteableBitSet interface and make a subclass of OpenBitSet that is writeable to disk.  We're working on an isSparse method for OpenBitSet.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org