You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Otis Gospodnetic (JIRA)" <ji...@apache.org> on 2006/12/11 22:42:21 UTC
[jira] Created: (LUCENE-741) Field norm modifier (CLI tool)
Field norm modifier (CLI tool)
------------------------------
Key: LUCENE-741
URL: http://issues.apache.org/jira/browse/LUCENE-741
Project: Lucene - Java
Issue Type: New Feature
Components: Index
Reporter: Otis Gospodnetic
Priority: Minor
I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-741?page=all ]
Otis Gospodnetic resolved LUCENE-741.
-------------------------------------
Resolution: Fixed
Committed. I'll also remove the old version of this code (+ its unit test), the one that still lives in contrib/miscellaneous/src/java/org/apache/lucene/misc/ .
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: http://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Otis Gospodnetic updated LUCENE-741:
------------------------------------
Attachment: LUCENE-741.patch
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464109 ]
Doron Cohen commented on LUCENE-741:
------------------------------------
Attached for.nrm.patch was very noisy - so I replaced it with one created with
svn diff -x --ignore-eol-style contrib/miscellaneous
It is relative to trunk.
A test is added to TestFieldNormModifier.java - testModifiedNormValuesCombinedWithKill - that verifies exactly what are the values of norms after modification.
FieldNormModifier modified to handle .nrm file as outlined above.
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: for.nrm.patch, LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464105 ]
Doron Cohen commented on LUCENE-741:
------------------------------------
I was looking at what it would take to make this work with .nrm file as well.
I expected there will be a test that fails currently, but there is none.
So I looked into the tests and the implementation and have a few questions:
(1) under contrib, FieldNormModifier and LengthNormModifier seem quite similar, right?
The first one sets with:
- reader.setNorm(d, fieldName,
- sim.encodeNorm(sim.lengthNorm(fieldName, termCounts[d])));
The latter with:
- byte norm = sim.encodeNorm(sim.lengthNorm(fieldName, termCounts[d]));
- reader.setNorm(d, fieldName, norm);
Do we need to keep both?
(2) TestFieldNormModifier.testFieldWithNoNorm() calls resetNorms() for a field that does not exist. Some work is done by the modifier to collect the term frequencies, and then reader.setNorm is called but it does nothing, because there are no norms. And indeed the test verifies that there are still no norms for this field. Confusing I think. For some reason I assumed that calling resetNorms() for a field that has none, would implicitly set omitNorms to false for that field and compute it - the inverse of killNorms(). Since this is not the case, perhaps resetNorms should throw an exception in this case?
(3) I would feel safer about this feature if the test was more strict - something like TestNorms - have several fields, modify some, each in a unique way, remove some others, then at the end verify that all the values of each field norms are exactly as expected.
(4) For killNorms to work, you can first revert the index to not use .nrm, and then "kill" as before. The code knows to read .fN files, for both backwards compatibility, and for reading segments created be DocumentWriter. The following steps will do this:
- read the norms using reader.norm(field)
- write into .fN files
- remove .nrm file
- modify segmentInfo to know that it has no .nrm file.
(5) It would have been more efficient to optimize (and remove the .nrm file) once in the application, so perhaps modify the public API to take an array of fields and operate on all?
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Otis Gospodnetic reopened LUCENE-741:
-------------------------------------
The norm-removing functionality was bogus - it simply "normalized the norms" to be 1 for the given field, but did not completely remove norms for a field, and did not flip the omitNorms bit for the given field, so it was never a true NO_NORMS field.
I'll upload a new patch that does this, but it does it only for Lucene 2.0.0 and Lucene 2.1-dev before the new .nrm changes from LUCENE-756 were committed.
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-741:
-------------------------------
Attachment: for.nrm.patch
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: for.nrm.patch, LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-741?page=all ]
Otis Gospodnetic reassigned LUCENE-741:
---------------------------------------
Assignee: Otis Gospodnetic
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: http://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-741:
-------------------------------
Attachment: (was: for.nrm.patch)
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-741?page=all ]
Otis Gospodnetic updated LUCENE-741:
------------------------------------
Lucene Fields: [Patch Available] (was: [New])
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: http://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-741?page=all ]
Otis Gospodnetic updated LUCENE-741:
------------------------------------
Attachment: LUCENE-741.patch
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: http://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Priority: Minor
> Attachments: LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-741) Field norm modifier (CLI tool)
Posted by "Doron Cohen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-741:
-------------------------------
Attachment: for.nrm.patch
> Field norm modifier (CLI tool)
> ------------------------------
>
> Key: LUCENE-741
> URL: https://issues.apache.org/jira/browse/LUCENE-741
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Index
> Reporter: Otis Gospodnetic
> Assigned To: Otis Gospodnetic
> Priority: Minor
> Attachments: for.nrm.patch, LUCENE-741.patch, LUCENE-741.patch
>
>
> I took Chris' LengthNormModifier (contrib/misc) and modified it slightly, to allow us to set fake norms on an existing fields, effectively making it equivalent to Field.Index.NO_NORMS.
> This is related to LUCENE-448 (NO_NORMS patch) and LUCENE-496 (LengthNormModifier contrib from Chris).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org