[jira] Commented: (LUCENE-2450) Explore write-once attr bindings in the analysis chain

["Robert Muir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917727#action_12917727 ] 

Robert Muir commented on LUCENE-2450:
-------------------------------------

What are our thoughts on devising a plan to move forward with this?

Can we produce a "not-so-sophisticated" backwards layer for our existing attributes model?
I think such a thing could really propel this forwards, for example we wouldn't need to deprecate the
existing APIs necessarily, or maybe over time, but we could at least offer an alternative.

At first when looking at this, admittedly I was a bit skeptical, but the more I think about it the more I like it.

There are big traps in the existing model, for example clearAttributes! I think pretty much every analysis
component that isn't in lucene itself that I've checked doesn't do this one right... and I'm talking simple 
ones (not ones that use context, etc)

I think in the short term we need to at least consider implementing LUCENE-2609 as an immediate solution
for the deficiencies in our current analysis system... in other words provide a toolkit for users who want
to write their own analysis components, to test them easily and avoid these traps.

But i'm starting to think this model is also more natural, and would make it easier for users to write their
own analysis components without having bugs... we are supposed to be a library and I think this is too
hard at the moment.





> Explore write-once attr bindings in the analysis chain
> ------------------------------------------------------
>
>                 Key: LUCENE-2450
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2450
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>            Reporter: Michael McCandless
>         Attachments: LUCENE-2450.patch, LUCENE-2450.patch, pipeline.py
>
>
> I'd like to propose a new means of tracking attrs through the analysis
> chain, whereby a given stage in the pipeline cannot overwrite attrs
> from stages before it (write once).  It can only write to new attrs
> (possibly w/ the same name) that future stages can see; it can never
> alter the attrs or bindings from the prior stages.
> I coded up a prototype chain in python (I'll attach), showing the
> equivalent of WhitespaceTokenizer -> StopFilter -> SynonymFilter ->
> Indexer.
> Each stage "sees" a frozen namespace of attr bindings as its input;
> these attrs are all read-only from its standpoint.  Then, it writes to
> an "output namespace", which is read/write, eg it can add new attrs,
> remove attrs from its input, change the values of attrs.  If that
> stage doesn't alter a given attr it "passes through", unchanged.
> This would be an enormous change to how attrs are managed... so this
> is very very exploratory at this point.  Once we decouple indexer from
> analysis, creating such an alternate chain should be possible -- it'd
> at least be a good test that we've decoupled enough :)
> I think the idea offers some compelling improvements over the "global
> read/write namespace" (AttrFactory) approach we have today:
>   * Injection filters can be more efficient -- they need not
>     capture/restoreState at all
>   * No more need for the initial tokenizer to "clear all attrs" --
>     each stage becomes responsible for clearing the attrs it "owns"
>   * You can truly stack stages (vs having to make a custom
>     AttrFactory) -- eg you could make a Bocu1 stage which can stack
>     onto any other stage.  It'd look up the CharTermAttr, remove it
>     from its output namespace, and add a BytesRefTermAttr.
>   * Indexer should be more efficient, in that it doesn't need to
>     re-get the attrs on each next() -- it gets them up front, and
>     re-uses them.
> Note that in this model, the indexer itself is just another stage in
> the pipeline, so you could do some wild things like use 2 indexer
> stages (writing to different indexes, or maybe the same index but
> somehow with further processing or something).
> Also, in this approach, the analysis chain is more informed about the
> what each stage is allowed to change, up front after the chain is
> created.  EG (say) we will know that only 2 stages write to the term
> attr, and that only 1 writes posIncr/offset attrs, etc.  Not sure
> if/how this helps us... but it's more strongly typed than what we have
> today.
> I think we could use a similar chain for processing a document at the
> field level, ie, different stages could add/remove/change different
> fields in the doc....

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org