You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Andrzej Bialecki (Created) (JIRA)" <ji...@apache.org> on 2012/01/03 23:08:39 UTC
[jira] [Created] (LUCENE-3672) IndexCommit.equals() bug
IndexCommit.equals() bug
------------------------
Key: LUCENE-3672
URL: https://issues.apache.org/jira/browse/LUCENE-3672
Project: Lucene - Java
Issue Type: Bug
Components: core/index
Affects Versions: 4.0
Reporter: Andrzej Bialecki
IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Michael McCandless (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3672:
---------------------------------------
Attachment: LUCENE-3672.patch
Patch.
I removed Directory.fileModified, and
IndexCommit.getVersion/getTimestamp. I changed Solr to take its own
timestamp (System.currentTimeMillis) and store into the commit's
userData, and fixed the places that needed it to look it up from
there. I also throw an exception if ever IndexCommit.compareTo is
passed an IndexCommit with a different Directory.
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
> Attachments: LUCENE-3672.patch
>
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179078#comment-13179078 ]
Robert Muir commented on LUCENE-3672:
-------------------------------------
and its even crazier because compareTo() uses generation.
if we move hashCode() and equals() to generation too, and completely remove IndexCommit.getVersion(), all tests pass in Lucene,
but Solr replication is using IndexCommit.getVersion() for some reason (is it a good reason?)
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179090#comment-13179090 ]
Robert Muir commented on LUCENE-3672:
-------------------------------------
{quote}
Hmmm, why is the version not bumped? That seems like the bug here (the index changed but the version didn't).
{quote}
Seems like version is pretty redundant here though, why not just get rid of it. the only thing using it is Solr, if it wants an additional integer here it could do this in the commit user data instead.
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Michael McCandless (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3672.
----------------------------------------
Resolution: Fixed
Fix Version/s: 4.0
3.6
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3672.patch
>
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179512#comment-13179512 ]
Robert Muir commented on LUCENE-3672:
-------------------------------------
i don't think we need any more timestamps.
Also the existing IndexCommit getTimeStamp should be removed.
We currently don't fsync the directory file itself and this is totally wrong on e.g. NFS
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179507#comment-13179507 ]
Yonik Seeley commented on LUCENE-3672:
--------------------------------------
Since bumping version and generation on changes is redundant, what if we replaced version with a creation timestamp?
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179503#comment-13179503 ]
Yonik Seeley commented on LUCENE-3672:
--------------------------------------
Generation isn't as good as version for telling if something has changed - it's much easier (probably trivial) to get a false match under real usage scenarios.
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Michael McCandless (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-3672:
------------------------------------------
Assignee: Michael McCandless
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
> Assignee: Michael McCandless
> Attachments: LUCENE-3672.patch
>
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179083#comment-13179083 ]
Yonik Seeley commented on LUCENE-3672:
--------------------------------------
bq. This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers.
Hmmm, why is the version not bumped? That seems like the bug here (the index changed but the version didn't).
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179480#comment-13179480 ]
Michael McCandless commented on LUCENE-3672:
--------------------------------------------
+1 to nuking version, and fixing equals/compareTo/hashCode to only use generation; I think version is redundant w/ the generation.
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3672) IndexCommit.equals() bug
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179084#comment-13179084 ]
Robert Muir commented on LUCENE-3672:
-------------------------------------
its also not ideal and that equals() is inconsistent with compareTo() even if we fixed it this way, because compareTo() doesn't consider Directory.
> IndexCommit.equals() bug
> ------------------------
>
> Key: LUCENE-3672
> URL: https://issues.apache.org/jira/browse/LUCENE-3672
> Project: Lucene - Java
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0
> Reporter: Andrzej Bialecki
>
> IndexCommit.equals() checks for equality of Directories and versions, but it doesn't check IMHO the more important generation numbers. It looks like commits are really identified by a combination of directory and segments_XXX, which means the generation number, because that's what the DirectoryReader.open() checks for.
> This bug leads to an unexpected behavior when the only change to be committed is in userData - we get two commits then that are declared equal, they have the same version but they have different generation numbers. I have no idea how this situation is treated in a few dozen references to IndexCommit.equals() across Lucene...
> On the surface the fix is trivial - either add the gen number to equals(), or use gen number instead of version. However, it's puzzling why these two would ever get out of sync??? and if they are always supposed to be in sync then maybe we don't need both of them at all, maybe just generation or version is sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org