You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Peter Karman <pe...@peknet.com> on 2011/09/09 05:25:59 UTC
[lucy-dev] Re: which fields contained which terms
Marvin Humphrey wrote on 8/30/11 4:59 PM:
> To support highlighting, at index-time we create an inverted representation
> for each field that has been marked as "highlightable", then serialize all the
> inverted fields together in one blob (called, for no particularly good reason,
> a "DocVector"). Effectively this is a miniature inverted-index containing a
> single document. The class which does the work is
> Lucy::Index::HighlightWriter, and the relevant segment files are named
> seg_NNN/highlight.ix and seg_NNN/highlight.dat.
I finally found time to try this, but I must be doing something wrong, because
despite setting all my fields to 'highlightable', no seg_NNN/highlight.* files
are getting created.
here's a snip from schema_1.json:
"fields" : {
"author" : {
"analyzer" : "1",
"highlightable" : "1",
"sortable" : "1",
"type" : "fulltext"
},
"dates" : {
"analyzer" : "1",
"highlightable" : "1",
"sortable" : "1",
"type" : "fulltext"
},
"orgs" : {
"analyzer" : "1",
"highlightable" : "1",
"sortable" : "1",
"type" : "fulltext"
},
and here's what my index dir looks like:
index.swish3]$ tree
.
|-- locks
|-- schema_1.json
|-- seg_1
| |-- cf.dat
| |-- cfmeta.json
| `-- segmeta.json
|-- snapshot_1.json
|-- swish.xml
`-- swish_last_start
2 directories, 7 files
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] Re: which fields contained which terms
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 9/9/11 12:45 AM:
> The highlight.* files exist as discrete files for a little while during
> indexing, but then they get rolled into the compound file.
ah! thanks.
/me continues with testing from original thread.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] Re: which fields contained which terms
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Thu, Sep 08, 2011 at 10:25:59PM -0500, Peter Karman wrote:
> Marvin Humphrey wrote on 8/30/11 4:59 PM:
>
> > To support highlighting, at index-time we create an inverted representation
> > for each field that has been marked as "highlightable", then serialize all the
> > inverted fields together in one blob (called, for no particularly good reason,
> > a "DocVector"). Effectively this is a miniature inverted-index containing a
> > single document. The class which does the work is
> > Lucy::Index::HighlightWriter, and the relevant segment files are named
> > seg_NNN/highlight.ix and seg_NNN/highlight.dat.
>
> I finally found time to try this, but I must be doing something wrong, because
> despite setting all my fields to 'highlightable', no seg_NNN/highlight.* files
> are getting created.
The highlight.* files exist as discrete files for a little while during
indexing, but then they get rolled into the compound file. Take a look in
seg_NNN/cfmeta.json.
> |-- locks
> |-- schema_1.json
> |-- seg_1
> | |-- cf.dat
> | |-- cfmeta.json
> | `-- segmeta.json
> |-- snapshot_1.json
> |-- swish.xml
> `-- swish_last_start
The "cf.dat" file has the data, and the "cfmeta.json" file contains the file
metadata, such as offset and length.
Marvin Humphrey