You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Miller (JIRA)" <ji...@apache.org> on 2009/11/16 22:12:39 UTC
[jira] Created: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Document issues involved in building your index with one jdk version and then searching/updating with another
-------------------------------------------------------------------------------------------------------------
Key: LUCENE-2073
URL: https://issues.apache.org/jira/browse/LUCENE-2073
Project: Lucene - Java
Issue Type: Task
Reporter: Mark Miller
I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
{quote}
If there is nothing we can do here, then we just have to do the best we can -
such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
{quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778882#action_12778882 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
bq. Which components are affected by this? I think just Analyzers and query parsers, yes?
I think just tokenstreams / analyzers.
bq. This isn't a fundamental problem with the Lucene architecture; it's an artifact of the way certain classes are implemented.
yes, you are exactly right. StandardTokenizer does not have this problem. its behavior is independent of the users JVM, although dependent on the JVM of the developer who last re-ran jflex :)
bq. Perhaps there could be an "ICUAnalysis" package which depends on ICU4J
yeah, this is what I think most of us who need unicode support do anyway. so i tried to start a good impl of this in LUCENE-1488.
even if you do not care about unicode support, maybe you care that things like .isWhiteSpace() are faster than in the jdk:
http://site.icu-project.org/home/why-use-icu4j (see under Performance section).
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779004#action_12779004 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Mark, but here is the crux of the issue:
While i think casing property is normative and should not change, new characters can be introduced with new casing properties.
This of course should not affect Simple/StopAnalyzer, but may affect StandardAnalyzer.
The reason is that StandardTokenizer contains hardcoded (sometimes oversized) ranges that may include some characters that were previously unassigned in Unicode 3.
If they are assigned in Unicode 4, with a casing property, then this means for lucene, they were indexed in uppercase in 1.4 but lowercase in 1.5... i hope this makes sense.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779020#action_12779020 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
{quote}
> I am pretty sure StandardAnalyzer is ok actually now.
Good news! Thanks for performing that analysis.
{quote}
Doh! this case pair stability policy was not introduced until 5.0, so to validate, i must do further analysis
ref: "unicode back compat policy": http://www.unicode.org/policies/stability_policy.html
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779030#action_12779030 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Hi, I completely validated. StandardAnalyzer is ok. none of the casing changed for any previous [:letter:] characters from Unicode 3 in Unicode 4.
I'll work on adding more explanation
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779010#action_12779010 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
Okay, fair enough - in the end I do think we want to err on the side of caution - losing terms/docs in the index can be disastrous for some. Can we contain the statement at all though? What if you are just working with essentially ascii text? Basic English chars? That type of thing ... perhaps saying something to the effect of, in many/some cases, a user might not be affected, but if you need to be sure, you must reindex?
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778886#action_12778886 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Hello, whereever we put this text (i do not care), I thought the following would be a start.
please help me make it easier to digest/understand
{noformat}
If possible, use the same major JVM version at both index and search time.
Different JVM versions may implement different versions of the Unicode Standard, which will change the way Lucene treats your text.
For example, under Java 1.4, LetterTokenizer will split around the character U+02C6, but under Java 5 it will not.
This is because Java 1.4 implements Unicode 3, but Java 5 implements Unicode 4.
{noformat}
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779015#action_12779015 ]
Marvin Humphrey commented on LUCENE-2073:
-----------------------------------------
> I am pretty sure StandardAnalyzer is ok actually now.
Good news! Thanks for performing that analysis.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779086#action_12779086 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
Looks great! +1
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779094#action_12779094 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Committed revision 881466.
should we keep the issue open Mark, or close it?
I think we will potentially have some specialized problems in 3.1 also... (due to fixing unicode 4/suppl characters)
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778547#action_12778547 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Mark, I agree, there are two issues I know of:
* using a new java version can change unicode version, which changes character properties, which makes things tokenize differently.
* using a different default locale can change at least some things in contrib, for example PatternAnalyzer uses a Locale-based toLowerCase() method, which will break if default locale is different between index and search.
... <other things you can think of>
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2073:
--------------------------------
Attachment: LUCENE-2073.patch
i tried to improve the wording a bit.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2073:
--------------------------------
Attachment: stdAnalyzerTest.java
the crappy test script i wrote to confirm the lowercasing behavior.
maybe we need it again if we go to java 1.7 in this decade, or maybe i screwed something up :)
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779018#action_12779018 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
bq. if you are using "Basic Latin" as exists on your keyboard, you might be ok.
Heh - well if thats the strongest you would said that, sounds like its not very containable. Perhaps we just put that?
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779012#action_12779012 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Mark good point, if you are using "Basic Latin" as exists on your keyboard, you might be ok.
Of course, one never knows, I feel like i remember Microsoft Word creating strange hyphens and smart quotes automatically...
I'd love to contain it, if you have some suggested verbage lets put it in there.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778999#action_12778999 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Hi Marvin, thanks for reviewing it.
{quote}
... but I still think the message is a little too aggressive. There are a lot
of people just using ye olde StandardAnalyzer, and they don't need to reindex.
We don't need to spread our own FUD.
{quote}
are you sure? StandardAnalyzer uses LowerCaseFilter, i did not check to see if any casing properties have changed.
{quote}
Can we change it to say "Analyzers", and then refer people to the docs for
their specific Analyzer? Alternatively, should that notification just contain
a complete list of the affected classes?
{quote}
Basically I am trying to be ambiguous, because I'm not sure what all is affected without doing some significant analysis!
I will do this, if its the right thing though.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778890#action_12778890 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
I guess that warning still isnt good enough. if you bump jvm version you should reindex older data too, and not just add new documents to it...
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778875#action_12778875 ]
Marvin Humphrey commented on LUCENE-2073:
-----------------------------------------
Which components are affected by this? I think just Analyzers and query
parsers, yes?
If that's true, my inclination would be to add a note to the javadocs for each
such class. In every case, it's theoretically possible to build alternative
implementations which are unaffected by upgrading the JVM.
This isn't a fundamental problem with the Lucene architecture; it's an
artifact of the way certain classes are implemented. Outside of the affected
components, Lucene doesn't get down and dirty with Unicode properties and
other fast-moving stuff -- it's just dealing in UTF-8 bytes, Java strings,
etc. Those things can change (Modified UTF-8, shudder), but they move on a
slower timescale.
Arguably, Analyzer subclasses shouldn't be in core for reasons like this.
Perhaps there could be an "ICUAnalysis" package which depends on ICU4J, so
that Unicode-related index incompatibilites occur when you upgrade your
Unicode library. Though most people would probably choose to use the
smaller-footprint, zero-dependency "JVMAnalysis" package, where reindexing
would be required after a JVM upgrade.
The software certifiers wouldn't like that, and I'm not seriously advocating
such a disruptive change (yet), but I just wanted to illustrate that this is a
contained problem.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779049#action_12779049 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Uwe, ok, I am currently trying to come up with a way to word this.
Unknowns like the soft hyphen thing and other things word processors, etc might do make me really cautious to say something like this...
I don't want someone to unnecessarily reindex, but on the other hand, I don't want to see bug report or whatever either.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-2073:
--------------------------------
Fix Version/s: 3.1
Okay - popped it to 3.1 to ensure we look at this again.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Fix For: 3.1
>
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2073) Document issues involved
in building your index with one jdk version and then searching/updating
with another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779018#action_12779018 ]
Mark Miller edited comment on LUCENE-2073 at 11/17/09 6:04 PM:
---------------------------------------------------------------
bq. if you are using "Basic Latin" as exists on your keyboard, you might be ok.
Heh - well if thats the strongest you would say that, sounds like its not very containable. Perhaps we just put that?
*edit*
Yes, word def uses a couple funky things by default with hyphens and quotes - ran into that before. But I think its somewhat up to you to know that. If you are getting your text from another application, than you don't likely properly know you have latin1 stuff. But some people will know - in the newspaper biz, we use pre processing filters to map anything outside of latin1 to latin1 - else remove it. If you don't know your source text well enough, then yes, all bets are off.
was (Author: markrmiller@gmail.com):
bq. if you are using "Basic Latin" as exists on your keyboard, you might be ok.
Heh - well if thats the strongest you would said that, sounds like its not very containable. Perhaps we just put that?
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778815#action_12778815 ]
Uwe Schindler commented on LUCENE-2073:
---------------------------------------
How about adding a general note into some JRE_VERSION_MIGRATION.txt on each release, also add this to the non-version specific site docs?
The first version that would contain it in the distribution would be 3.0, but it should be noted on the site-docs, that this affects *all* versions. Something like "whenever you change JRE major versions 1.4 -> 1.5 -> 1.6 -> 1.7 you may have to reindex, because the underlying VM's Unicode version could have changed in a way, affecting your tokenizers."
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778862#action_12778862 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
Sounds good to me.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778998#action_12778998 ]
Marvin Humphrey commented on LUCENE-2073:
-----------------------------------------
I like this:
> some parts of Lucene
... but I still think the message is a little too aggressive. There are a lot
of people just using ye olde StandardAnalyzer, and they don't need to reindex.
We don't need to spread our own FUD. :)
Can we change it to say "Analyzers", and then refer people to the docs for
their specific Analyzer? Alternatively, should that notification just contain
a complete list of the affected classes?
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2073:
--------------------------------
Attachment: LUCENE-2073.patch
ok, i think this one is better.
For convenience, I list which JRE versions correspond to which unicode versions.
Because I think that for example, if someone goes from Java 5 to Java 6, they should not worry about this.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779023#action_12779023 ]
Uwe Schindler commented on LUCENE-2073:
---------------------------------------
bq. Of course, one never knows, I feel like i remember Microsoft Word creating strange hyphens and smart quotes automatically...
It is the same strange soft hyphen we talked about before... Normally you should simply remove it during indexing using a CharFilter (I do this when indexing word documents and so on).
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779083#action_12779083 ]
Uwe Schindler commented on LUCENE-2073:
---------------------------------------
+1, ok
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779098#action_12779098 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
Cool, we can do something similar in 3.1
While it seems crazy, like these issues affect no one, someone told me today about relying upon broken unicode 4 behavior in lucene for chinese tokenization....!
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Fix For: 3.1
>
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779011#action_12779011 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
I am pretty sure StandardAnalyzer is ok actually now.
The only time it uses hardcoded ranges is for the CJK definition:
and the following UnicodeSet is the empty set (0 codepoints):
{noformat}
[[\u3100-\u312f\u3040-\u309F\u30A0-\u30FF\u31F0-\u31FF\u3300-\u337f\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uff65-\uff9f]&[:Case_Sensitive=True:]]
{noformat}
WikipediaTokenizer, probably not ok :)
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2073:
--------------------------------
Attachment: LUCENE-2073.patch
i added the following:
{quote}
In general, whether or not you need to re-index largely depends upon the data that
you are searching, and what was changed in any given Unicode version. For example,
if you are completely sure that your content is limited to the "Basic Latin" range
of Unicode, you can safely ignore this.
{quote}
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2073) Document issues involved in building
your index with one jdk version and then searching/updating with another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2073:
--------------------------------
Attachment: LUCENE-2073.patch
very happy to add new verbage here mentioning that StandardAnalyzer is ok.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779000#action_12779000 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
Agreed - we don't want to cause those that don't need to to reindex just to try and be safe.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778880#action_12778880 ]
Mark Miller commented on LUCENE-2073:
-------------------------------------
{quote} If that's true, my inclination would be to add a note to the javadocs for each
such class. In every case, it's theoretically possible to build alternative
implementations which are unaffected by upgrading the JVM. {quote}
I think thats a good idea, but I'd still like the top level warning as well - I think it should be specific in explaining the problem, and not claim that it exists for everything - Robert has already laid out a couple specific examples we can mention. The problem with the javadoc approach only is that its very easy to miss - I think its a good place to lay out the details for that particular class - but it would still be great to have a visible top level warning so that users know to be careful of this.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779043#action_12779043 ]
Uwe Schindler commented on LUCENE-2073:
---------------------------------------
Text looks good, maybe we should still add the "basic latin as on your keyboard".
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779078#action_12779078 ]
Robert Muir commented on LUCENE-2073:
-------------------------------------
guys I would like to commit this file soon, let me know if you see any issues with it.
I think this additional verbage i added about "basic latin" is safe, the only properties that have changed even up to unicode 5.1 are minor changes to bidirectional properties.
This only affects rendering/layout.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, LUCENE-2073.patch, stdAnalyzerTest.java
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2073) Document issues involved in
building your index with one jdk version and then searching/updating with
another
Posted by "Marvin Humphrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779006#action_12779006 ]
Marvin Humphrey commented on LUCENE-2073:
-----------------------------------------
> are you sure? StandardAnalyzer uses LowerCaseFilter,
No, I'm not sure. :( I was confusing StandardAnalyzer and StandardTokenizer.
I still think that there are a lot of people who don't need to reindex,
because, for example, their entire corpus is limited to Latin-1 code points.
Conversely, the people most likely to be affected are the people most likely
to be on the lookout for this kind of think. I think it's important to
reach this group, without unduly alarming those who don't really need to
reindex. Reindexing is a huge pain for some installations.
> Document issues involved in building your index with one jdk version and then searching/updating with another
> -------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-2073
> URL: https://issues.apache.org/jira/browse/LUCENE-2073
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Mark Miller
> Attachments: LUCENE-2073.patch, LUCENE-2073.patch
>
>
> I think this needs to go in something of a permenant spot - this isn't a one time release type issues - its going to present over multiple release.
> {quote}
> If there is nothing we can do here, then we just have to do the best we can -
> such as a prominent notice alerting that if you transition JVM's between building and searching the index and you are using or doing X, things will break.
> We should put this in a spot that is always pretty visible - perhaps even a new readme file titlted something like IndexBackwardCompatibility or something, to which we can add other tips and gotchyas as they come up. Or MaintainingIndicesAcrossVersions, or FancyWhateverGetsYourAttentionAboutUpgradingStuff. Or a permanent entry/sticky entry at the top of Changes.
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org