You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Grant Ingersoll (JIRA)" <ji...@apache.org> on 2007/12/04 17:13:44 UTC
[jira] Created: (LUCENE-1077) Analysis Sinks package
Analysis Sinks package
----------------------
Key: LUCENE-1077
URL: https://issues.apache.org/jira/browse/LUCENE-1077
Project: Lucene - Java
Issue Type: New Feature
Components: Analysis, contrib/*
Reporter: Grant Ingersoll
Assignee: Grant Ingersoll
Priority: Minor
Fix For: 2.3
With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1077) New Analysis Contributions
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated LUCENE-1077:
------------------------------------
Summary: New Analysis Contributions (was: Analysis Sinks package)
Expanding to add in some other analysis capabilities
> New Analysis Contributions
> ---------------------------
>
> Key: LUCENE-1077
> URL: https://issues.apache.org/jira/browse/LUCENE-1077
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis, contrib/*
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1077) New Analysis Contributions
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated LUCENE-1077:
------------------------------------
Attachment: LUCENE-1077.patch
Thought of another possibly useful payload convenience TokenFilter that adds the Token type as the payload. I've always wondered why we don't make more use of the type attribute on a Token.
> New Analysis Contributions
> ---------------------------
>
> Key: LUCENE-1077
> URL: https://issues.apache.org/jira/browse/LUCENE-1077
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis, contrib/*
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1077.patch, LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1077) New Analysis Contributions
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll resolved LUCENE-1077.
-------------------------------------
Resolution: Fixed
Lucene Fields: (was: [New])
Committed
> New Analysis Contributions
> ---------------------------
>
> Key: LUCENE-1077
> URL: https://issues.apache.org/jira/browse/LUCENE-1077
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis, contrib/*
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1077.patch, LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1077) Analysis Sinks package
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated LUCENE-1077:
------------------------------------
Attachment: LUCENE-1077.patch
This is a fairly trivial start to to this, but it creates the sinks package in the contrib/Analysis section and adds a simple TokenRangeSinkTokenizer and test. This can be used to siphon off tokens that fall in a range. All it does is count the tokens that go by and add those that fall in the range. It might be useful for documents that you know have certain structures. For instance, if you know the first 5 tokens of your docs are X.
More to follow.
> Analysis Sinks package
> ----------------------
>
> Key: LUCENE-1077
> URL: https://issues.apache.org/jira/browse/LUCENE-1077
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis, contrib/*
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1077) New Analysis Contributions
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated LUCENE-1077:
------------------------------------
Attachment: LUCENE-1077.patch
Adds to the sinks package:
DateRecognizerSinkTokenizer which only adds dates that can be parsed by a DateFormat object
TokenRangeSinkTokenizer as described earlier
TokenTypeSinkTokenizer only adds to the sink if the token type is a specific value.
Adds the payloads package, which contains
NumericPayloadTokenFilter -- Assigns a predefined float-based payload to a Token if the type matches the specified input type of the Token. As a use case, this could be used to assign a payload for all tokens that are marked as "bold" or some other value.
> New Analysis Contributions
> ---------------------------
>
> Key: LUCENE-1077
> URL: https://issues.apache.org/jira/browse/LUCENE-1077
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Analysis, contrib/*
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing. See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org