You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Grant Ingersoll (JIRA)" <ji...@apache.org> on 2007/12/04 17:13:44 UTC

[jira] Created: (LUCENE-1077) Analysis Sinks package

Analysis Sinks package
----------------------

                 Key: LUCENE-1077
                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
             Project: Lucene - Java
          Issue Type: New Feature
          Components: Analysis, contrib/*
            Reporter: Grant Ingersoll
            Assignee: Grant Ingersoll
            Priority: Minor
             Fix For: 2.3


With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.

This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1077) New Analysis Contributions

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll updated LUCENE-1077:
------------------------------------

    Summary: New Analysis  Contributions  (was: Analysis Sinks package)

Expanding to add in some other analysis capabilities

> New Analysis  Contributions
> ---------------------------
>
>                 Key: LUCENE-1077
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis, contrib/*
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1077) New Analysis Contributions

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll updated LUCENE-1077:
------------------------------------

    Attachment: LUCENE-1077.patch

Thought of another possibly useful payload convenience TokenFilter that adds the Token type as the payload.  I've always wondered why we don't make more use of the type attribute on a Token.

> New Analysis  Contributions
> ---------------------------
>
>                 Key: LUCENE-1077
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis, contrib/*
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-1077.patch, LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Resolved: (LUCENE-1077) New Analysis Contributions

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll resolved LUCENE-1077.
-------------------------------------

       Resolution: Fixed
    Lucene Fields:   (was: [New])

Committed

> New Analysis  Contributions
> ---------------------------
>
>                 Key: LUCENE-1077
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis, contrib/*
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-1077.patch, LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1077) Analysis Sinks package

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll updated LUCENE-1077:
------------------------------------

    Attachment: LUCENE-1077.patch

This is a fairly trivial start to to this, but it creates the sinks package in the contrib/Analysis section and adds a simple TokenRangeSinkTokenizer and test.  This can be used to siphon off tokens that fall in a range.  All it does is count the tokens that go by and add those that fall in the range.  It might be useful for documents that you know have certain structures.  For instance, if you know the first 5 tokens of your docs are X.

More to follow.

> Analysis Sinks package
> ----------------------
>
>                 Key: LUCENE-1077
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis, contrib/*
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


[jira] Updated: (LUCENE-1077) New Analysis Contributions

Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-1077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Grant Ingersoll updated LUCENE-1077:
------------------------------------

    Attachment: LUCENE-1077.patch

Adds to the sinks package:
DateRecognizerSinkTokenizer which only adds dates that can be parsed by a DateFormat object

TokenRangeSinkTokenizer as described earlier

TokenTypeSinkTokenizer only adds to the sink if the token type is a specific value.

Adds the payloads package, which contains
NumericPayloadTokenFilter -- Assigns a predefined float-based payload to a Token if the type matches the specified input type of the Token.  As a use case, this could be used to assign a payload for all tokens that are marked as "bold" or some other value.

> New Analysis  Contributions
> ---------------------------
>
>                 Key: LUCENE-1077
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1077
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis, contrib/*
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: LUCENE-1077.patch, LUCENE-1077.patch
>
>
> With the advent of the new TeeTokenFilter and SinkTokenizer, there now exists some interesting new things that can be done in the analysis phase of indexing.  See LUCENE-1058.
> This patch provides some new implementations of SinkTokenizer that may be useful.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org