Analyzers thread safe across indexes?

[Bill Gray <bi...@10gen.com.INVALID>]

Hi,

I'm working on an app that will use Lucene and I'm trying to get a handle
of the thread safety of Analyzers.

The specific question I'm exploring is whether it is safe to use a single
instance of an Analyzer across different Indexes (and therefore concurrent
IndexWriters) in the case where two indexes may include the same
Analyzer/fieldName pair?

Our app will create and use multiple indexes concurrently and it is likely
that several of those indexes will contain the same field and specify the
same Analyzer.

It seems like the ReuseStrategy is keyed off Analyzer and fieldName.  So
the Tokenizers would get reused across indexes.  It is not clear to me that
all the Tokenizers are not stateful, but if they are not stateful, then I
guess it would be good to reuse a single Analyzer instance across indexes
in this way?

Thank you.

Bill

Re: Analyzers thread safe across indexes?

[Bill Gray <bi...@10gen.com.INVALID>]

Thank you Adrien.

On Thu, Jan 24, 2019 at 10:10 PM Adrien Grand <jp...@gmail.com> wrote:

> Hi Bill,
>
> Yes, reusing analyzers across different indexes is safe.
>
> Tokenizers (and some token filters and char filters) are stateful, but
> state caching is performed in a ThreadLocal so no state is shared
> between threads.
>
> On Fri, Jan 25, 2019 at 1:04 AM Bill Gray <bi...@10gen.com.invalid>
> wrote:
> >
> > Hi,
> >
> > I'm working on an app that will use Lucene and I'm trying to get a handle
> > of the thread safety of Analyzers.
> >
> > The specific question I'm exploring is whether it is safe to use a single
> > instance of an Analyzer across different Indexes (and therefore
> concurrent
> > IndexWriters) in the case where two indexes may include the same
> > Analyzer/fieldName pair?
> >
> > Our app will create and use multiple indexes concurrently and it is
> likely
> > that several of those indexes will contain the same field and specify the
> > same Analyzer.
> >
> > It seems like the ReuseStrategy is keyed off Analyzer and fieldName.  So
> > the Tokenizers would get reused across indexes.  It is not clear to me
> that
> > all the Tokenizers are not stateful, but if they are not stateful, then I
> > guess it would be good to reuse a single Analyzer instance across indexes
> > in this way?
> >
> > Thank you.
> >
> > Bill
>
>
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Analyzers thread safe across indexes?

[Adrien Grand <jp...@gmail.com>]

Hi Bill,

Yes, reusing analyzers across different indexes is safe.

Tokenizers (and some token filters and char filters) are stateful, but
state caching is performed in a ThreadLocal so no state is shared
between threads.

On Fri, Jan 25, 2019 at 1:04 AM Bill Gray <bi...@10gen.com.invalid> wrote:
>
> Hi,
>
> I'm working on an app that will use Lucene and I'm trying to get a handle
> of the thread safety of Analyzers.
>
> The specific question I'm exploring is whether it is safe to use a single
> instance of an Analyzer across different Indexes (and therefore concurrent
> IndexWriters) in the case where two indexes may include the same
> Analyzer/fieldName pair?
>
> Our app will create and use multiple indexes concurrently and it is likely
> that several of those indexes will contain the same field and specify the
> same Analyzer.
>
> It seems like the ReuseStrategy is keyed off Analyzer and fieldName.  So
> the Tokenizers would get reused across indexes.  It is not clear to me that
> all the Tokenizers are not stateful, but if they are not stateful, then I
> guess it would be good to reuse a single Analyzer instance across indexes
> in this way?
>
> Thank you.
>
> Bill



-- 
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org