You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Scott Guthery <sg...@gmail.com> on 2022/09/22 22:58:45 UTC
Max Field Length
Lucene 9.3 seems to have a (post-Analyzer) maximum field length of 32767.
Is there a way of increasing this without resorting to the source code?
Thanks for any guidance.
Cheers, Scott
Re: Max Field Length
Posted by Michael Sokolov <ms...@gmail.com>.
ooh
On Fri, Sep 23, 2022 at 11:02 AM Adrien Grand <jp...@gmail.com> wrote:
>
> We have a TruncateTokenFilter in lucene/analysis/common. :)
>
> On Fri, Sep 23, 2022 at 4:39 PM Michael Sokolov <ms...@gmail.com> wrote:
>
> > I wonder if it would make sense to provide a TruncationFilter in
> > addition to the LengthFilter. That way long tokens in source text
> > could be better supported, albeit with some confusion if they share
> > the same very long prefix...
> >
> > On Fri, Sep 23, 2022 at 9:56 AM Scott Guthery <sg...@gmail.com> wrote:
> > >
> > > Thanks much, Adrian. I hadn't realized that the size limit was on one
> > > token in the text as opposed to being a limit on the length of the entire
> > > text field. I'm loading patents, so I suspect that the very long word
> > is a
> > > DNA sequence.
> > >
> > > Thanks also for your guidance with regard to setting maximums.
> > >
> > > Cheers, Scott
> > >
> > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> --
> Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Max Field Length
Posted by Adrien Grand <jp...@gmail.com>.
We have a TruncateTokenFilter in lucene/analysis/common. :)
On Fri, Sep 23, 2022 at 4:39 PM Michael Sokolov <ms...@gmail.com> wrote:
> I wonder if it would make sense to provide a TruncationFilter in
> addition to the LengthFilter. That way long tokens in source text
> could be better supported, albeit with some confusion if they share
> the same very long prefix...
>
> On Fri, Sep 23, 2022 at 9:56 AM Scott Guthery <sg...@gmail.com> wrote:
> >
> > Thanks much, Adrian. I hadn't realized that the size limit was on one
> > token in the text as opposed to being a limit on the length of the entire
> > text field. I'm loading patents, so I suspect that the very long word
> is a
> > DNA sequence.
> >
> > Thanks also for your guidance with regard to setting maximums.
> >
> > Cheers, Scott
> >
> > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Adrien
Re: Max Field Length
Posted by Michael Sokolov <ms...@gmail.com>.
I wonder if it would make sense to provide a TruncationFilter in
addition to the LengthFilter. That way long tokens in source text
could be better supported, albeit with some confusion if they share
the same very long prefix...
On Fri, Sep 23, 2022 at 9:56 AM Scott Guthery <sg...@gmail.com> wrote:
>
> Thanks much, Adrian. I hadn't realized that the size limit was on one
> token in the text as opposed to being a limit on the length of the entire
> text field. I'm loading patents, so I suspect that the very long word is a
> DNA sequence.
>
> Thanks also for your guidance with regard to setting maximums.
>
> Cheers, Scott
>
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Max Field Length
Posted by Scott Guthery <sg...@gmail.com>.
Thanks much, Adrian. I hadn't realized that the size limit was on one
token in the text as opposed to being a limit on the length of the entire
text field. I'm loading patents, so I suspect that the very long word is a
DNA sequence.
Thanks also for your guidance with regard to setting maximums.
Cheers, Scott
>
>
Re: Max Field Length
Posted by Adrien Grand <jp...@gmail.com>.
Hi Scott,
There is no way to lift this limit. The assumption is that a user would
never type a 32kB keyword in a search bar, so indexing such long keywords
is wasteful. Some tokenizers like StandardTokenizer can be configured to
limit the length of the tokens that they produce, there is also a
LengthFilter that can be appended to the analysis chain to filter out
tokens that exceed the maximum term length.
I would note that modifying the source code is going to require more than
bumping the hardcoded limit as we rely on this limit in a few places, e.g.
ByteBlockPool.
On Fri, Sep 23, 2022 at 12:59 AM Scott Guthery <sg...@gmail.com> wrote:
> Lucene 9.3 seems to have a (post-Analyzer) maximum field length of 32767.
> Is there a way of increasing this without resorting to the source code?
>
> Thanks for any guidance.
>
> Cheers, Scott
>
--
Adrien