[jira] Commented: (LUCENE-830) norms file can become unexpectedly enormous

["Doron Cohen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12480520 ] 

Doron Cohen commented on LUCENE-830:
------------------------------------

> One simple workaround is to disable norms. 

You mean for some of the fields, using Fieldable's setOmitNorms().

For large indexes, I would think that most fields would be indexed with omit=true, except for one (content) or two (subject?) fields were length normalization and/or boosting are of importance. in such cases there would not really be a problem.

Consider the example that an index created for adding textual search to a database application, by mapping the index field names to the database "textual columns" names; if more than one table is indexed, but the textual column name happens to be different between the tables, then yes, - with that straightforward mapping there would be a waste - lots of unused bytes. 

One work around for such applications could be to map the textual columns of all tables to a single textual field in Lucene, thuogh then they would have to filter by a table-name field (which they might do anyhow). 


> norms file can become unexpectedly enormous
> -------------------------------------------
>
>                 Key: LUCENE-830
>                 URL: https://issues.apache.org/jira/browse/LUCENE-830
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Index
>    Affects Versions: 2.1
>            Reporter: Michael McCandless
>            Priority: Minor
>
> Spinoff from this user thread:
>    http://www.gossamer-threads.com/lists/lucene/java-user/46754
> Norms are not stored sparsely, so even if a doc doesn't have field X
> we still use up 1 byte in the norms file (and in memory when that
> field is searched) for that segment.  I think this is done for
> performance at search time?
> For indexes that have a large # documents where each document can have
> wildly varying fields, each segment will use # documents times # fields
> seen in that segment.  When optimize merges all segments, that product
> grows multiplicatively so the norms file for the single segment will
> require far more storage than the sum of all previous segments' norm
> files.
> I think it's uncommon to have a huge number of distinct fields (?) so
> we would need a solution that doesn't hurt the more common case where
> most documents have the same fields.  Maybe something analogous to how
> bitvectors are now optionally stored sparsely?
> One simple workaround is to disable norms.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org