You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Nicolas Lalevée (JIRA)" <ji...@apache.org> on 2008/01/25 11:16:34 UTC
[jira] Created: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
The token types of the standard tokenizer is not accessible
-----------------------------------------------------------
Key: LUCENE-1150
URL: https://issues.apache.org/jira/browse/LUCENE-1150
Project: Lucene - Java
Issue Type: Bug
Components: Analysis
Affects Versions: 2.3
Reporter: Nicolas Lalevée
The StandardTokenizerImpl not being public, these token types are not accessible :
{code:java}
public static final int ALPHANUM = 0;
public static final int APOSTROPHE = 1;
public static final int ACRONYM = 2;
public static final int COMPANY = 3;
public static final int EMAIL = 4;
public static final int HOST = 5;
public static final int NUM = 6;
public static final int CJ = 7;
/**
* @deprecated this solves a bug where HOSTs that end with '.' are identified
* as ACRONYMs. It is deprecated and will be removed in the next
* release.
*/
public static final int ACRONYM_DEP = 8;
public static final String [] TOKEN_TYPES = new String [] {
"<ALPHANUM>",
"<APOSTROPHE>",
"<ACRONYM>",
"<COMPANY>",
"<EMAIL>",
"<HOST>",
"<NUM>",
"<CJ>",
"<ACRONYM_DEP>"
};
{code}
So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1150:
------------------------------------------
Assignee: Michael McCandless
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Reopened: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reopened LUCENE-1150:
----------------------------------------
You're right. I'll put that back as well, and port to 2.3.2.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Fix For: 2.3.2, 2.4
>
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562525#action_12562525 ]
Michael McCandless commented on LUCENE-1150:
--------------------------------------------
Good! I'll take that approach, and update WikipediaTokenizer too.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Attachments: LUCENE-1150.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1150:
---------------------------------------
Attachment: LUCENE-1150.take2.patch
New patch attached, that also exposes the token types for WikipediaTokenizer. I'll commit in a day or two.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1150.
----------------------------------------
Resolution: Fixed
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Fix For: 2.3.2, 2.4
>
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Nicolas Lalevée (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562424#action_12562424 ]
Nicolas Lalevée commented on LUCENE-1150:
-----------------------------------------
here is my workaround until it is fixed into the Lucene API :
{code:java}
package org.apache.lucene.analysis.standard;
public class TokenTypeAccessor {
public static final String APOSTROPHE_TYPE = StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.APOSTROPHE];
public static final String ACRONYM_TYPE = StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.ACRONYM];
public static final String HOST_TYPE = StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.HOST];
}
{code}
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1150:
---------------------------------------
Fix Version/s: 2.3.2
Backported fix to 2.3.2.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Fix For: 2.3.2, 2.4
>
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Antony Bowesman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588953#action_12588953 ]
Antony Bowesman commented on LUCENE-1150:
-----------------------------------------
The original tokenImage String array from 2.2 is still not available in this patch, they are still in the Impl. These are the values returned from Token.type(), so should they not be visible as well as the static ints?
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Fix For: 2.3.2, 2.4
>
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562491#action_12562491 ]
Grant Ingersoll commented on LUCENE-1150:
-----------------------------------------
Why not just add them on to the StandardTokenizer class?
For the WikipediaTokenizer (roughly based on the StandardTokenizer), I just added them to the WikipediaTokenizer wrapper class. However, I did leave the StandardTokenizer ones as they were. So, we should probably do the appropriate thing there, too.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Attachments: LUCENE-1150.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1150.
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.4
I just committed this. Thanks for opening this Nicolas!
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Fix For: 2.4
>
> Attachments: LUCENE-1150.patch, LUCENE-1150.take2.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1150:
---------------------------------------
Attachment: LUCENE-1150.patch
Attached patch fixing this. I just added a new Constants.java that has static constants defined, and added a compile-time testcase to assert that these constants remain publicly accessible.
I will commit in a day or two.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
> Assignee: Michael McCandless
> Attachments: LUCENE-1150.patch
>
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1150) The token types of the standard
tokenizer is not accessible
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562472#action_12562472 ]
Michael McCandless commented on LUCENE-1150:
--------------------------------------------
Ugh, I missed that we lost this when we switched to JFlex (LUCENE-966). I'll take this.
> The token types of the standard tokenizer is not accessible
> -----------------------------------------------------------
>
> Key: LUCENE-1150
> URL: https://issues.apache.org/jira/browse/LUCENE-1150
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Affects Versions: 2.3
> Reporter: Nicolas Lalevée
>
> The StandardTokenizerImpl not being public, these token types are not accessible :
> {code:java}
> public static final int ALPHANUM = 0;
> public static final int APOSTROPHE = 1;
> public static final int ACRONYM = 2;
> public static final int COMPANY = 3;
> public static final int EMAIL = 4;
> public static final int HOST = 5;
> public static final int NUM = 6;
> public static final int CJ = 7;
> /**
> * @deprecated this solves a bug where HOSTs that end with '.' are identified
> * as ACRONYMs. It is deprecated and will be removed in the next
> * release.
> */
> public static final int ACRONYM_DEP = 8;
> public static final String [] TOKEN_TYPES = new String [] {
> "<ALPHANUM>",
> "<APOSTROPHE>",
> "<ACRONYM>",
> "<COMPANY>",
> "<EMAIL>",
> "<HOST>",
> "<NUM>",
> "<CJ>",
> "<ACRONYM_DEP>"
> };
> {code}
> So no custom TokenFilter can be based of the token type. Actually even the StandardFilter cannot be writen outside the org.apache.lucene.analysis.standard package.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org