[jira] Issue Comment Edited: (LUCENE-2186) First cut at column-stride fields (index values storage)

["Simon Willnauer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896122#action_12896122 ] 

Simon Willnauer edited comment on LUCENE-2186 at 8/7/10 1:09 PM:
-----------------------------------------------------------------

Hey Yonik,
bq. Could you show some examples of the most efficient way to use this API?

Sure! While it's already late over here I am happy to provide you those two examples. This is how you can index CSF with this Attribute approach:

{code} 
    RAMDirectory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(Version.LUCENE_40, new SimpleAnalyzer(Version.LUCENE_40)));
    Document doc = new Document();
    Fieldable fieldable = new AttributeField("myIntField");
    ValuesAttribute valuesAttribute = fieldable.attributes().addAttribute(ValuesAttribute.class);
    valuesAttribute.setType(Values.PACKED_INTS);
    valuesAttribute.ints().set(100);
    doc.add(fieldable);    
    writer.addDocument(doc);
    writer.close();
{code}
  
This is how get the values back via source or the ValuesEnum:

{code}
    IndexReader reader = IndexReader.open(dir);
    // this might be integrated into Fields eventually
    Reader indexValues = reader.getIndexValues("myIntField"); // can get cached version too via reader.getIndexValuesCache();
    Source load = indexValues.load();
    long value = load.ints(0);
    System.out.println(value);

    // or get it from the enum
    ValuesEnum intEnum = indexValues.getEnum();
    ValuesAttribute attr = intEnum.getAttribute(ValuesAttribute.class);
    while(intEnum.nextDoc() != ValuesEnum.NO_MORE_DOCS) {
      System.out.println(attr.ints().get());
    }
{code}

I guess this should make it at least easier to get started. I actually expect people saying {{ValuesEnum}} looks very much like {{DocsEnum}} which is certainly correct. Yet, I didn't integrate {{ValuesEnum}} into {{Fields}} etc. for simplicity as changes to {{Fields}} touches lot of code. Having {{ValuesEnum}} being a "stand-alone" API makes iterating and development easier and a cut over to DocsEnum would be easy API wise as it already implements {{DocIdSetIterator}}. 

With that in mind, the use of {{ValuesAttribute}} makes sense too - with a stand-also API this would be obsolet and could be direct part of {{ValuesEnum}}.

What I don't like about {{DocsEnum}} is the {{BulkReadResult}} class and its relative being a first class citizen in {{DocsEnum}}. With CSF not every {{DocsEnum}} iterates over <id,freq>* - but maybe we can move that to an attribute and make a more general abstract class.  The Attributes on those enums don't introduce a real overhead but solve lots of extendability problems though.

      was (Author: simonw):
    Hey Yonik,
bq. Could you show some examples of the most efficient way to use this API?

Sure! While it's already late over here I am happy to provide you those two examples. This is how you can index CSF with this Attribute approach:

{code}
    RAMDirectory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(Version.LUCENE_40, new SimpleAnalyzer(Version.LUCENE_40)));
    Document doc = new Document();
    Fieldable fieldable = new AttributeField("myIntField");
    ValuesAttribute valuesAttribute = fieldable.attributes().addAttribute(ValuesAttribute.class);
    valuesAttribute.setType(Values.PACKED_INTS);
    valuesAttribute.ints().set(100);
    doc.add(fieldable);    
    writer.addDocument(doc);
    writer.close();
{code}
  
This is how get the values back via source or the ValuesEnum:

{code}
    IndexReader reader = IndexReader.open(dir);
    // this might be integrated into Fields eventually
    Reader indexValues = reader.getIndexValues("myIntField"); // can get cached version too via reader.getIndexValuesCache();
    Source load = indexValues.load();
    long value = load.ints(0);
    System.out.println(value);

    // or get it from the enum
    ValuesEnum intEnum = indexValues.getEnum();
    ValuesAttribute attr = intEnum.getAttribute(ValuesAttribute.class);
    while(intEnum.nextDoc() != ValuesEnum.NO_MORE_DOCS) {
      System.out.println(attr.ints().get());
    }
{code}

I guess this should make it at least easier to get started.
  
> First cut at column-stride fields (index values storage)
> --------------------------------------------------------
>
>                 Key: LUCENE-2186
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2186
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael McCandless
>            Assignee: Simon Willnauer
>             Fix For: 4.0
>
>         Attachments: LUCENE-2186.patch, LUCENE-2186.patch, LUCENE-2186.patch, LUCENE-2186.patch, mem.py
>
>
> I created an initial basic impl for storing "index values" (ie
> column-stride value storage).  This is still a work in progress... but
> the approach looks compelling.  I'm posting my current status/patch
> here to get feedback/iterate, etc.
> The code is standalone now, and lives under new package
> oal.index.values (plus some util changes, refactorings) -- I have yet
> to integrate into Lucene so eg you can mark that a given Field's value
> should be stored into the index values, sorting will use these values
> instead of field cache, etc.
> It handles 3 types of values:
>   * Six variants of byte[] per doc, all combinations of fixed vs
>     variable length, and stored either "straight" (good for eg a
>     "title" field), "deref" (good when many docs share the same value,
>     but you won't do any sorting) or "sorted".
>   * Integers (variable bit precision used as necessary, ie this can
>     store byte/short/int/long, and all precisions in between)
>   * Floats (4 or 8 byte precision)
> String fields are stored as the UTF8 byte[].  This patch adds a
> BytesRef, which does the same thing as flex's TermRef (we should merge
> them).
> This patch also adds basic initial impl of PackedInts (LUCENE-1990);
> we can swap that out if/when we get a better impl.
> This storage is dense (like field cache), so it's appropriate when the
> field occurs in all/most docs.  It's just like field cache, except the
> reading API is a get() method invocation, per document.
> Next step is to do basic integration with Lucene, and then compare
> sort performance of this vs field cache.
> For the "sort by String value" case, I think RAM usage & GC load of
> this index values API should be much better than field caache, since
> it does not create object per document (instead shares big long[] and
> byte[] across all docs), and because the values are stored in RAM as
> their UTF8 bytes.
> There are abstract Writer/Reader classes.  The current reader impls
> are entirely RAM resident (like field cache), but the API is (I think)
> agnostic, ie, one could make an MMAP impl instead.
> I think this is the first baby step towards LUCENE-1231.  Ie, it
> cannot yet update values, and the reading API is fully random-access
> by docID (like field cache), not like a posting list, though I
> do think we should add an iterator() api (to return flex's DocsEnum)
> -- eg I think this would be a good way to track avg doc/field length
> for BM25/lnu.ltc scoring.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org