You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2009/09/03 23:29:00 UTC

[jira] Created: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

remove deprecated custom encoding support in russian/greek analysis
-------------------------------------------------------------------

                 Key: SOLR-1410
                 URL: https://issues.apache.org/jira/browse/SOLR-1410
             Project: Solr
          Issue Type: Task
          Components: Analysis
            Reporter: Robert Muir


In this case, analyzers have strange encoding support and it has been deprecated in lucene.

For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ

LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
    Analyzers. If you need to index text in these encodings, please use Java's
    character set conversion facilities (InputStreamReader, etc) during I/O, 
    so that Lucene can analyze this text as Unicode instead.

I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0

Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated SOLR-1410:
---------------------------

    Fix Version/s: 1.4
         Assignee: Hoss Man

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761481#action_12761481 ] 

Shalin Shekhar Mangar commented on SOLR-1410:
---------------------------------------------

bq. Will there be a Solr release based on Lucene 3.0, or will 1.5 be based on 3.1?

I guess it is too early to say. But Solr releases do take time so if I had to guess it is likely that 1.5 will go out with Lucene 3.1

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751180#action_12751180 ] 

Robert Muir commented on SOLR-1410:
-----------------------------------

thanks, I will work on a patch later that logs a warning if you try to use a configuration option for anything but Unicode for these two analyzers!


> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated SOLR-1410:
------------------------------

    Attachment: SOLR-1410.patch

for russian and greek analysis factories, warn users if they try to use the deprecated charset parameter.

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752890#action_12752890 ] 

Hoss Man commented on SOLR-1410:
--------------------------------

Committed revision 812760.

thanks robert

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751177#action_12751177 ] 

Hoss Man commented on SOLR-1410:
--------------------------------

I don't think we've ever really had a situation like this ...logging a warning seems like the right course of action for now ... then once the functionality is removed, we can change the factory to fail on init if it sees the option is still set in the schema.xml

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man resolved SOLR-1410.
----------------------------

    Resolution: Fixed

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761309#action_12761309 ] 

Robert Muir commented on SOLR-1410:
-----------------------------------

Hi, I just removed these deprecations for Lucene 3.0 (which does not affect 1.4)

However, in doing so I noticed that with the custom charset removed, RussianLowerCaseFilter is really exactly the same as LowerCaseFilter.
I've marked this RussianLowerCaseFilter as deprecated to be removed in Lucene 3.1

Will there be a Solr release based on Lucene 3.0, or will 1.5 be based on 3.1?


> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751374#action_12751374 ] 

Shalin Shekhar Mangar commented on SOLR-1410:
---------------------------------------------

bq. I don't think we've ever really had a situation like this ...logging a warning seems like the right course of action for now ... 

We actually have done this in DataImportHandler in relation to the syntax for evaluators. Logging a warning is the right way to go.


> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Muir updated SOLR-1410:
------------------------------

    Priority: Minor  (was: Major)

> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752868#action_12752868 ] 

Robert Muir commented on SOLR-1410:
-----------------------------------

are there any issues with this... care if i set version 1.4?

really hoping to remove these pseudo-charsets after the lucene 2.9 release :)


> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Priority: Minor
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (SOLR-1410) remove deprecated custom encoding support in russian/greek analysis

Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761593#action_12761593 ] 

Robert Muir commented on SOLR-1410:
-----------------------------------

ok, I guess anyway this isn't an issue. 
if 1.5 goes out with 3.1, RussianLowerCaseFilterFactory can be implemented with LowerCaseFilter, but marked deprecated to be removed in 1.6


> remove deprecated custom encoding support in russian/greek analysis
> -------------------------------------------------------------------
>
>                 Key: SOLR-1410
>                 URL: https://issues.apache.org/jira/browse/SOLR-1410
>             Project: Solr
>          Issue Type: Task
>          Components: Analysis
>            Reporter: Robert Muir
>            Assignee: Hoss Man
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: SOLR-1410.patch
>
>
> In this case, analyzers have strange encoding support and it has been deprecated in lucene.
> For example someone using CP1251 in the russian analyzer is simply storing Ж as 0xC6, its being represented as Æ
> LUCENE-1793: Deprecate the custom encoding support in the Greek and Russian
>     Analyzers. If you need to index text in these encodings, please use Java's
>     character set conversion facilities (InputStreamReader, etc) during I/O, 
>     so that Lucene can analyze this text as Unicode instead.
> I noticed in solr, the factories for these tokenstreams allow these configuration options, which are deprecated in 2.9 to be removed in 3.0
> Let me know the policy (how do you deprecate a config option in solr exactly, log a warning, etc?) and I'd be happy to create a patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.