You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Peter Karman <pe...@peknet.com> on 2011/09/09 05:25:59 UTC

[lucy-dev] Re: which fields contained which terms

Marvin Humphrey wrote on 8/30/11 4:59 PM:

> To support highlighting, at index-time we create an inverted representation
> for each field that has been marked as "highlightable", then serialize all the
> inverted fields together in one blob (called, for no particularly good reason,
> a "DocVector").  Effectively this is a miniature inverted-index containing a
> single document.  The class which does the work is
> Lucy::Index::HighlightWriter, and the relevant segment files are named
> seg_NNN/highlight.ix and seg_NNN/highlight.dat.

I finally found time to try this, but I must be doing something wrong, because
despite setting all my fields to 'highlightable', no seg_NNN/highlight.* files
are getting created.

here's a snip from schema_1.json:

   "fields" : {
      "author" : {
         "analyzer" : "1",
         "highlightable" : "1",
         "sortable" : "1",
         "type" : "fulltext"
      },
      "dates" : {
         "analyzer" : "1",
         "highlightable" : "1",
         "sortable" : "1",
         "type" : "fulltext"
      },
      "orgs" : {
         "analyzer" : "1",
         "highlightable" : "1",
         "sortable" : "1",
         "type" : "fulltext"
      },

and here's what my index dir looks like:

index.swish3]$ tree
.
|-- locks
|-- schema_1.json
|-- seg_1
|   |-- cf.dat
|   |-- cfmeta.json
|   `-- segmeta.json
|-- snapshot_1.json
|-- swish.xml
`-- swish_last_start

2 directories, 7 files


-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] Re: which fields contained which terms

Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 9/9/11 12:45 AM:

> The highlight.* files exist as discrete files for a little while during
> indexing, but then they get rolled into the compound file.

ah! thanks.

/me continues with testing from original thread.

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-dev] Re: which fields contained which terms

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Thu, Sep 08, 2011 at 10:25:59PM -0500, Peter Karman wrote:
> Marvin Humphrey wrote on 8/30/11 4:59 PM:
> 
> > To support highlighting, at index-time we create an inverted representation
> > for each field that has been marked as "highlightable", then serialize all the
> > inverted fields together in one blob (called, for no particularly good reason,
> > a "DocVector").  Effectively this is a miniature inverted-index containing a
> > single document.  The class which does the work is
> > Lucy::Index::HighlightWriter, and the relevant segment files are named
> > seg_NNN/highlight.ix and seg_NNN/highlight.dat.
> 
> I finally found time to try this, but I must be doing something wrong, because
> despite setting all my fields to 'highlightable', no seg_NNN/highlight.* files
> are getting created.

The highlight.* files exist as discrete files for a little while during
indexing, but then they get rolled into the compound file.  Take a look in
seg_NNN/cfmeta.json.

> |-- locks
> |-- schema_1.json
> |-- seg_1
> |   |-- cf.dat
> |   |-- cfmeta.json
> |   `-- segmeta.json
> |-- snapshot_1.json
> |-- swish.xml
> `-- swish_last_start

The "cf.dat" file has the data, and the "cfmeta.json" file contains the file
metadata, such as offset and length.

Marvin Humphrey