[jira] Commented: (LUCENE-1292) Tag Index

["Jason Rutherglen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615615#action_12615615 ] 

Jason Rutherglen commented on LUCENE-1292:
------------------------------------------

I decided to rework this to not use a transaction log.  The reason being the transaction log should be global and be per segment.  The other reason is the current architecture used the transaction log to read from.  Given the small size of postings, I decided it would be best to keep the changed postings blocks in memory.  This way there is no performance hit from the realtime update feature.  When the RAM usage hits a specified number then a new term + postings file can be written.  

One issue is how to implement skipTo over the blocks.  The postings blocks use the SkipListWriter implementation.  However the question is what is the best way to skip over the blocks themselves?  One approach is to use the modified binary search from InstantiatedIndex described at http://ochafik.free.fr/blog/?p=106.  The other is to implement a skip list over the blocks, however I am not sure SkipListWriter will work given it is tied to using file pointers.  If SkipListWriter were used, would it be best to have it skip over the blocks using only the maxDoc for each block?  This is the approach the binarysearch method would use. 

At the beginning of each term's list of posting blocks, there is a set of BlockInfos describing the blocks.  Some blocks are missing because they don't have postings.  Blocks are predetermined to span a document number range rather than a length in bytes range.  This is to not have to deal with document number overlap issues between blocks when an update occurs.  Right now I am thinking the default block size should be 4000 docs.  This should yield a block size in bytes of a little over 4000 at the maximum.  It is unclear how the size of the block will affect the performance of the skip process.  

> Tag Index
> ---------
>
>                 Key: LUCENE-1292
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1292
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>         Attachments: lucene-1292.patch
>
>
> The problem the tag index solves is slow field cache loading and range queries, and reindexing an entire document to update fields that are not tokenized.  
> The tag index holds untokenized terms with a docfreq of 1 in a term dictionary like index file.  The file also stores the docs per term, similar to LUCENE-1278.  The index also has a transaction log and in memory index for realtime updates to the tags.  The transaction log is periodically merged into the existing tag term dictionary index file.
> The TagIndexReader extends IndexReader and is unified with a regular index by ParallelReader.  There is a doc id to terms skip pointer file for the IndexReader.document method.  This file contains a pointer for looking up the terms for a document.  
> There is a higher level class that encapsulates writing a document with tag fields to IndexWriter and TagIndexWriter.  This requires a hook into IndexWriter to coordinate doc ids and flushing segments to disk.  
> The writer class could be as simple as:
> {code}
> public class TagIndexWriter {
>   
>   public void add(Term term, DocIdSetIterator iterator) {
>   }
>   
>   public void delete(Term term, DocIdSetIterator iterator) {
>   }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org