You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2012/11/06 02:30:12 UTC
[jira] [Created] (LUCENE-4540) Allow packed ints norms
Robert Muir created LUCENE-4540:
-----------------------------------
Summary: Allow packed ints norms
Key: LUCENE-4540
URL: https://issues.apache.org/jira/browse/LUCENE-4540
Project: Lucene - Core
Issue Type: Task
Components: core/index
Reporter: Robert Muir
Attachments: LUCENE-4540.patch
I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491391#comment-13491391 ]
Michael McCandless commented on LUCENE-4540:
--------------------------------------------
+1, very cool!
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491185#comment-13491185 ]
Robert Muir commented on LUCENE-4540:
-------------------------------------
I tried this on that geonames database since my default indexing (just shoving everything in as a TextField)
creates a huge .nrm file today (150MB: 8M docs * 19 fields). Just as a test I tried a simple similarity
implementation that uses
{code}
@Override
public void computeNorm(FieldInvertState state, Norm norm) {
norm.setPackedLong(state.getLength());
}
{code}
{noformat}
-rw-rw-r-- 1 rmuir rmuir 49339454 Nov 5 22:30 _7e_nrm.cfs
{noformat}
If you want to use boosts too, you would have to be careful how you encode, but I think this can be useful.
In this case its 1/3 of the RAM, even though documents lengths are exact vs. lossy (though most fields are
shortish, some are huge, like alternate names fields for major countries and cities, which have basically every
language imaginable shoved in the field: thats why it doesnt save more I think)
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-4540.
---------------------------------
Resolution: Fixed
Fix Version/s: 5.0
4.1
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Fix For: 4.1, 5.0
>
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491415#comment-13491415 ]
Robert Muir commented on LUCENE-4540:
-------------------------------------
I don't understand the question Simon: all the ones we provide happen to use Norm.setByte
I don't think we need to add documentation to Norm.setFloat,Norm.setDouble saying that we don't
provide any similarities that call these methods: thats not important to anybody.
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491417#comment-13491417 ]
Simon Willnauer commented on LUCENE-4540:
-----------------------------------------
bq. I don't understand the question Simon: all the ones we provide happen to use Norm.setByte
Just to clarify. Currently if we write packed ints and a similarity calls Source#getArray you get an UOE. I think we should document that our current impls won't handle this.
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491422#comment-13491422 ]
Simon Willnauer commented on LUCENE-4540:
-----------------------------------------
fair enough. I just wanted to mention it..
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491423#comment-13491423 ]
Robert Muir commented on LUCENE-4540:
-------------------------------------
If someone changes their similarity to use a different norm type at index-time than at query-time,
then he or she is an idiot!
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491421#comment-13491421 ]
Robert Muir commented on LUCENE-4540:
-------------------------------------
I don't see how its relevant. Issues will happen if you use Norm.setFloat (as they expect a byte).
I'm not going to confuse the documentation. The built-in Similarities at query-time
depend upon their index-time norm implementation: this is documented extensively everywhere!
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-4540) Allow packed ints norms
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-4540:
--------------------------------
Attachment: LUCENE-4540.patch
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-4540) Allow packed ints norms
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491341#comment-13491341 ]
Simon Willnauer commented on LUCENE-4540:
-----------------------------------------
+1 - should we also document that we don't have similarities that can make use of it at this point?
> Allow packed ints norms
> -----------------------
>
> Key: LUCENE-4540
> URL: https://issues.apache.org/jira/browse/LUCENE-4540
> Project: Lucene - Core
> Issue Type: Task
> Components: core/index
> Reporter: Robert Muir
> Attachments: LUCENE-4540.patch
>
>
> I was curious what the performance would be, because it might be useful option to use packedints for norms if you have lots of fields and still want good scoring:
> Today the smallest norm per-field-per-doc you can use is a single byte, and if you have _f_ fields with norms enabled and _n_ docs, it uses _f_ * _n_ bytes of space in RAM.
> Especially if you aren't using index-time boosting (or even if you are, but not with ridiculous values), this could be wasting a ton of RAM.
> But then I noticed there was no clean way to allow you to do this in your Similarity: its a trivial patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org