You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Charles Sanders <cs...@redhat.com> on 2015/05/15 16:49:18 UTC
Problem with solr.LengthFilterFactory
I'm seeing a problem with the LengthFilter. It appears to work fine until I increase the max value above 254. At the point it stops removing the very large token from the stream. As a result I get the error:
java.lang.IllegalArgumentException: Document contains at least one immense term...... UTF8 encoding is longer than the max length 32766
I'm certain I'm doing this wrong. Can someone please show me the light. :)
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="254" />
</analyzer>
</fieldType>
Solr Version - 4.8.1
-Charles
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Shawn,
Sorry about my mistake. I have rerun the test with the correct fieldType definition. Still have the same results. See test information below.
<field name="portal_package" type="text_std" indexed="true" stored="true" multiValued="true"/>
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="300" />
</analyzer>
</fieldType>
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
}
{
"responseHeader": {
"status": 0,
"QTime": 43,
"params": {
"indent": "true",
"q": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"_": "1431951254019",
"wt": "json"
}
},
"response": {
"numFound": 1,
"start": 0,
"docs": [
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": [
"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
],
"_version_": 1501509688204722200,
"timestamp": "2015-05-18T12:13:45.464Z",
"language": "en"
}
]
}
}
----- Original Message -----
From: "Shawn Heisey" <ap...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 4:22:10 PM
Subject: Re: Problem with solr.LengthFilterFactory
On 5/15/2015 2:06 PM, Charles Sanders wrote:
> I have deleted all records from my index. Stopped solr. Made the schema changes as requested. Started solr. Then insert the one test record. Then search. Still see the same results. No portal_package is not the unique key, its uri. Which is a string field.
>
> <field name="portal_package" type="text_std" indexed="true" stored="true" multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </fieldType>
You got rid of <analyzer> and it's closing tag entirely, that's not
going to work. This is what you need, and you'll need to reindex the
doc again:
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="300"/>
</analyzer>
</fieldType>
Here's a temporary paste that you can copy/paste:
http://apaste.info/NXn
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Jack Krupansky <ja...@gmail.com>.
Just for our future reference:
1. Which schema field type analyzer combination was producing the exception?
2. Can you provide the full stack trace for the exception?
It shouldn't be possible to get the exception when using the white space
tokenizer, due to the noted issue, so which tokenizer was it?
-- Jack Krupansky
On Mon, May 18, 2015 at 12:21 PM, Charles Sanders <cs...@redhat.com>
wrote:
> No, the field has always been text. And from the error, its obviously
> passing a very large token to the index, regardless of the tokenizer and
> filter.
>
> So I guess I will have to tokenize and filter the text before I send it to
> solr, since solr is not able to properly handle a very large token. More
> custom code.
>
> Thanks everyone for your help.
> Charles
>
>
> ----- Original Message -----
>
> From: "Jack Krupansky" <ja...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Monday, May 18, 2015 12:00:37 PM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> Sorry for not spotting that earlier. Lucene itself does have such a limit.
> No way around it - an individual term is limited to 32K-2 bytes. Lucene is
> designed for searching of terms, not large blob storage.
>
> Maybe you defined that field as a string originally and later updated your
> schema to a text field, so that Lucene still knows it as an unanalyzed
> string field? You need to delete the index and start over if you want to
> change the field types like that.
>
> -- Jack Krupansky
>
> On Mon, May 18, 2015 at 8:33 AM, Charles Sanders <cs...@redhat.com>
> wrote:
>
> > Jack,
> > Thanks for the information. If I understand this correctly, the White
> > space tokenizer will break a single token of size 300 into two tokens,
> one
> > of size 256 and the other of size 44. If this is true, then for the
> single
> > test document I have used, in the index in the portal_package field, I
> > should see two tokens rather than one large single token.
> >
> > If my understanding is correct, then why in my production system, where
> we
> > occasionally get a single very large token, do I see this error?
> > Caused by: java.lang.IllegalArgumentException: Document contains at least
> > one immense term in field="portal_package" (whose UTF8 encoding is longer
> > than the max length 32766)
> >
> > The existence of this error would lead me to conclude that a very large
> > single token is making its way through the white space tokenizer and
> > filters to the index where it is rejected.
> >
> > I'm afraid my understanding is not complete. Can you fill in the gaps?
> >
> > Thanks,
> > Charles
> >
> >
> > ----- Original Message -----
> >
> > From: "Jack Krupansky" <ja...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Friday, May 15, 2015 4:31:22 PM
> > Subject: Re: Problem with solr.LengthFilterFactory
> >
> > Sorry that my brain has turned to mush... the issue you are hitting is
> due
> > to a known, undocumented limit in the whitespace tokenizer:
> >
> > https://issues.apache.org/jira/browse/LUCENE-5785
> > "White space tokenizer has undocumented limit of 256 characters per
> token"
> >
> > If you look at the parsed query you will see that two query terms were
> > generated. This is because the whitespace tokenizer will simply split
> long
> > tokens every 256 characters. So, your filter will never see a long term.
> >
> > There is a note on the Jira (evidently by me!) that you can use the
> pattern
> > tokenizer as a workaround. But... if your term is a string anyway, you
> > could just use the keyword tokenizer.
> >
> >
> > -- Jack Krupansky
> >
> > On Fri, May 15, 2015 at 4:06 PM, Charles Sanders <cs...@redhat.com>
> > wrote:
> >
> > > Shawn,
> > > Thanks a bunch for working with me on this.
> > >
> > > I have deleted all records from my index. Stopped solr. Made the schema
> > > changes as requested. Started solr. Then insert the one test record.
> Then
> > > search. Still see the same results. No portal_package is not the unique
> > > key, its uri. Which is a string field.
> > >
> > > <field name="portal_package" type="text_std" indexed="true"
> stored="true"
> > > multiValued="true"/>
> > >
> > > <fieldType name="text_std" class="solr.TextField"
> > > positionIncrementGap="100">
> > > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > > <filter class="solr.LengthFilterFactory" min="1" max="300" />
> > > </fieldType>
> > >
> > > {
> > > "documentKind": "test",
> > > "uri": "test300",
> > > "id": "test300",
> > >
> >
> "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > > }
> > >
> > >
> > > {
> > > "responseHeader": {
> > > "status": 0,
> > > "QTime": 47,
> > > "params": {
> > > "spellcheck": "true",
> > > "enableElevation": "false",
> > > "df": "allText",
> > > "echoParams": "all",
> > > "spellcheck.maxCollations": "5",
> > > "spellcheck.dictionary": "andreasAutoComplete",
> > > "spellcheck.count": "5",
> > > "spellcheck.collate": "true",
> > > "spellcheck.onlyMorePopular": "true",
> > > "rows": "10",
> > > "indent": "true",
> > > "q":
> > >
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > > "_": "1431719989047",
> > > "debug": "query",
> > > "wt": "json"
> > > }
> > > },
> > > "response": {
> > > "numFound": 1,
> > > "start": 0,
> > > "docs": [
> > > {
> > > "documentKind": "test",
> > > "uri": "test300",
> > > "id": "test300",
> > > "portal_package": [
> > >
> > >
> >
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > > ],
> > > "_version_": 1501267024421060600,
> > > "timestamp": "2015-05-15T19:56:43.247Z",
> > > "language": "en"
> > > }
> > > ]
> > > },
> > > "debug": {
> > > "rawquerystring":
> > >
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > > "querystring":
> > >
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > > "parsedquery":
> > >
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> > >
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > > "parsedquery_toString":
> > >
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> > >
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > > "QParser": "LuceneQParser"
> > > }
> > > }
> > >
> > >
> > >
> > >
> > >
> > > ----- Original Message -----
> > >
> > > From: "Shawn Heisey" <ap...@elyograg.org>
> > > To: solr-user@lucene.apache.org
> > > Sent: Friday, May 15, 2015 3:29:19 PM
> > > Subject: Re: Problem with solr.LengthFilterFactory
> > >
> > > On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> > > > Then I looked back at your fieldType definition and noticed that you
> > > > are only defining an index analyzer. Remove the 'type="index"' part
> of
> > > > the analyzer config so it happens at both index and query time,
> > > > reindex, then try again.
> > >
> > > The reindex may be very important here. I would actually completely
> > > delete your data directory and restart Solr before reindexing, to be
> > > sure you don't have old recordsfrom any previous reindexes.
> > >
> > > http://wiki.apache.org/solr/HowToReindex
> > >
> > > I think this next part is unlikely, but I'm going to ask it anyway: Is
> > > the portal_package field your schema uniqueKey? If it is, that might be
> > > an additional source of problems. Using a solr.Textfield for a
> > > uniqueKey field causes Solr to behave in unexpected ways.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> > >
> >
> >
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
No, the field has always been text. And from the error, its obviously passing a very large token to the index, regardless of the tokenizer and filter.
So I guess I will have to tokenize and filter the text before I send it to solr, since solr is not able to properly handle a very large token. More custom code.
Thanks everyone for your help.
Charles
----- Original Message -----
From: "Jack Krupansky" <ja...@gmail.com>
To: solr-user@lucene.apache.org
Sent: Monday, May 18, 2015 12:00:37 PM
Subject: Re: Problem with solr.LengthFilterFactory
Sorry for not spotting that earlier. Lucene itself does have such a limit.
No way around it - an individual term is limited to 32K-2 bytes. Lucene is
designed for searching of terms, not large blob storage.
Maybe you defined that field as a string originally and later updated your
schema to a text field, so that Lucene still knows it as an unanalyzed
string field? You need to delete the index and start over if you want to
change the field types like that.
-- Jack Krupansky
On Mon, May 18, 2015 at 8:33 AM, Charles Sanders <cs...@redhat.com>
wrote:
> Jack,
> Thanks for the information. If I understand this correctly, the White
> space tokenizer will break a single token of size 300 into two tokens, one
> of size 256 and the other of size 44. If this is true, then for the single
> test document I have used, in the index in the portal_package field, I
> should see two tokens rather than one large single token.
>
> If my understanding is correct, then why in my production system, where we
> occasionally get a single very large token, do I see this error?
> Caused by: java.lang.IllegalArgumentException: Document contains at least
> one immense term in field="portal_package" (whose UTF8 encoding is longer
> than the max length 32766)
>
> The existence of this error would lead me to conclude that a very large
> single token is making its way through the white space tokenizer and
> filters to the index where it is rejected.
>
> I'm afraid my understanding is not complete. Can you fill in the gaps?
>
> Thanks,
> Charles
>
>
> ----- Original Message -----
>
> From: "Jack Krupansky" <ja...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 4:31:22 PM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> Sorry that my brain has turned to mush... the issue you are hitting is due
> to a known, undocumented limit in the whitespace tokenizer:
>
> https://issues.apache.org/jira/browse/LUCENE-5785
> "White space tokenizer has undocumented limit of 256 characters per token"
>
> If you look at the parsed query you will see that two query terms were
> generated. This is because the whitespace tokenizer will simply split long
> tokens every 256 characters. So, your filter will never see a long term.
>
> There is a note on the Jira (evidently by me!) that you can use the pattern
> tokenizer as a workaround. But... if your term is a string anyway, you
> could just use the keyword tokenizer.
>
>
> -- Jack Krupansky
>
> On Fri, May 15, 2015 at 4:06 PM, Charles Sanders <cs...@redhat.com>
> wrote:
>
> > Shawn,
> > Thanks a bunch for working with me on this.
> >
> > I have deleted all records from my index. Stopped solr. Made the schema
> > changes as requested. Started solr. Then insert the one test record. Then
> > search. Still see the same results. No portal_package is not the unique
> > key, its uri. Which is a string field.
> >
> > <field name="portal_package" type="text_std" indexed="true" stored="true"
> > multiValued="true"/>
> >
> > <fieldType name="text_std" class="solr.TextField"
> > positionIncrementGap="100">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.LengthFilterFactory" min="1" max="300" />
> > </fieldType>
> >
> > {
> > "documentKind": "test",
> > "uri": "test300",
> > "id": "test300",
> >
> "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > }
> >
> >
> > {
> > "responseHeader": {
> > "status": 0,
> > "QTime": 47,
> > "params": {
> > "spellcheck": "true",
> > "enableElevation": "false",
> > "df": "allText",
> > "echoParams": "all",
> > "spellcheck.maxCollations": "5",
> > "spellcheck.dictionary": "andreasAutoComplete",
> > "spellcheck.count": "5",
> > "spellcheck.collate": "true",
> > "spellcheck.onlyMorePopular": "true",
> > "rows": "10",
> > "indent": "true",
> > "q":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "_": "1431719989047",
> > "debug": "query",
> > "wt": "json"
> > }
> > },
> > "response": {
> > "numFound": 1,
> > "start": 0,
> > "docs": [
> > {
> > "documentKind": "test",
> > "uri": "test300",
> > "id": "test300",
> > "portal_package": [
> >
> >
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > ],
> > "_version_": 1501267024421060600,
> > "timestamp": "2015-05-15T19:56:43.247Z",
> > "language": "en"
> > }
> > ]
> > },
> > "debug": {
> > "rawquerystring":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "querystring":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "parsedquery":
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "parsedquery_toString":
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "QParser": "LuceneQParser"
> > }
> > }
> >
> >
> >
> >
> >
> > ----- Original Message -----
> >
> > From: "Shawn Heisey" <ap...@elyograg.org>
> > To: solr-user@lucene.apache.org
> > Sent: Friday, May 15, 2015 3:29:19 PM
> > Subject: Re: Problem with solr.LengthFilterFactory
> >
> > On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> > > Then I looked back at your fieldType definition and noticed that you
> > > are only defining an index analyzer. Remove the 'type="index"' part of
> > > the analyzer config so it happens at both index and query time,
> > > reindex, then try again.
> >
> > The reindex may be very important here. I would actually completely
> > delete your data directory and restart Solr before reindexing, to be
> > sure you don't have old recordsfrom any previous reindexes.
> >
> > http://wiki.apache.org/solr/HowToReindex
> >
> > I think this next part is unlikely, but I'm going to ask it anyway: Is
> > the portal_package field your schema uniqueKey? If it is, that might be
> > an additional source of problems. Using a solr.Textfield for a
> > uniqueKey field causes Solr to behave in unexpected ways.
> >
> > Thanks,
> > Shawn
> >
> >
> >
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Jack Krupansky <ja...@gmail.com>.
Sorry for not spotting that earlier. Lucene itself does have such a limit.
No way around it - an individual term is limited to 32K-2 bytes. Lucene is
designed for searching of terms, not large blob storage.
Maybe you defined that field as a string originally and later updated your
schema to a text field, so that Lucene still knows it as an unanalyzed
string field? You need to delete the index and start over if you want to
change the field types like that.
-- Jack Krupansky
On Mon, May 18, 2015 at 8:33 AM, Charles Sanders <cs...@redhat.com>
wrote:
> Jack,
> Thanks for the information. If I understand this correctly, the White
> space tokenizer will break a single token of size 300 into two tokens, one
> of size 256 and the other of size 44. If this is true, then for the single
> test document I have used, in the index in the portal_package field, I
> should see two tokens rather than one large single token.
>
> If my understanding is correct, then why in my production system, where we
> occasionally get a single very large token, do I see this error?
> Caused by: java.lang.IllegalArgumentException: Document contains at least
> one immense term in field="portal_package" (whose UTF8 encoding is longer
> than the max length 32766)
>
> The existence of this error would lead me to conclude that a very large
> single token is making its way through the white space tokenizer and
> filters to the index where it is rejected.
>
> I'm afraid my understanding is not complete. Can you fill in the gaps?
>
> Thanks,
> Charles
>
>
> ----- Original Message -----
>
> From: "Jack Krupansky" <ja...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 4:31:22 PM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> Sorry that my brain has turned to mush... the issue you are hitting is due
> to a known, undocumented limit in the whitespace tokenizer:
>
> https://issues.apache.org/jira/browse/LUCENE-5785
> "White space tokenizer has undocumented limit of 256 characters per token"
>
> If you look at the parsed query you will see that two query terms were
> generated. This is because the whitespace tokenizer will simply split long
> tokens every 256 characters. So, your filter will never see a long term.
>
> There is a note on the Jira (evidently by me!) that you can use the pattern
> tokenizer as a workaround. But... if your term is a string anyway, you
> could just use the keyword tokenizer.
>
>
> -- Jack Krupansky
>
> On Fri, May 15, 2015 at 4:06 PM, Charles Sanders <cs...@redhat.com>
> wrote:
>
> > Shawn,
> > Thanks a bunch for working with me on this.
> >
> > I have deleted all records from my index. Stopped solr. Made the schema
> > changes as requested. Started solr. Then insert the one test record. Then
> > search. Still see the same results. No portal_package is not the unique
> > key, its uri. Which is a string field.
> >
> > <field name="portal_package" type="text_std" indexed="true" stored="true"
> > multiValued="true"/>
> >
> > <fieldType name="text_std" class="solr.TextField"
> > positionIncrementGap="100">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.LengthFilterFactory" min="1" max="300" />
> > </fieldType>
> >
> > {
> > "documentKind": "test",
> > "uri": "test300",
> > "id": "test300",
> >
> "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > }
> >
> >
> > {
> > "responseHeader": {
> > "status": 0,
> > "QTime": 47,
> > "params": {
> > "spellcheck": "true",
> > "enableElevation": "false",
> > "df": "allText",
> > "echoParams": "all",
> > "spellcheck.maxCollations": "5",
> > "spellcheck.dictionary": "andreasAutoComplete",
> > "spellcheck.count": "5",
> > "spellcheck.collate": "true",
> > "spellcheck.onlyMorePopular": "true",
> > "rows": "10",
> > "indent": "true",
> > "q":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "_": "1431719989047",
> > "debug": "query",
> > "wt": "json"
> > }
> > },
> > "response": {
> > "numFound": 1,
> > "start": 0,
> > "docs": [
> > {
> > "documentKind": "test",
> > "uri": "test300",
> > "id": "test300",
> > "portal_package": [
> >
> >
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> > ],
> > "_version_": 1501267024421060600,
> > "timestamp": "2015-05-15T19:56:43.247Z",
> > "language": "en"
> > }
> > ]
> > },
> > "debug": {
> > "rawquerystring":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "querystring":
> >
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "parsedquery":
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "parsedquery_toString":
> >
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> >
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> > "QParser": "LuceneQParser"
> > }
> > }
> >
> >
> >
> >
> >
> > ----- Original Message -----
> >
> > From: "Shawn Heisey" <ap...@elyograg.org>
> > To: solr-user@lucene.apache.org
> > Sent: Friday, May 15, 2015 3:29:19 PM
> > Subject: Re: Problem with solr.LengthFilterFactory
> >
> > On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> > > Then I looked back at your fieldType definition and noticed that you
> > > are only defining an index analyzer. Remove the 'type="index"' part of
> > > the analyzer config so it happens at both index and query time,
> > > reindex, then try again.
> >
> > The reindex may be very important here. I would actually completely
> > delete your data directory and restart Solr before reindexing, to be
> > sure you don't have old recordsfrom any previous reindexes.
> >
> > http://wiki.apache.org/solr/HowToReindex
> >
> > I think this next part is unlikely, but I'm going to ask it anyway: Is
> > the portal_package field your schema uniqueKey? If it is, that might be
> > an additional source of problems. Using a solr.Textfield for a
> > uniqueKey field causes Solr to behave in unexpected ways.
> >
> > Thanks,
> > Shawn
> >
> >
> >
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Jack,
Thanks for the information. If I understand this correctly, the White space tokenizer will break a single token of size 300 into two tokens, one of size 256 and the other of size 44. If this is true, then for the single test document I have used, in the index in the portal_package field, I should see two tokens rather than one large single token.
If my understanding is correct, then why in my production system, where we occasionally get a single very large token, do I see this error?
Caused by: java.lang.IllegalArgumentException: Document contains at least one immense term in field="portal_package" (whose UTF8 encoding is longer than the max length 32766)
The existence of this error would lead me to conclude that a very large single token is making its way through the white space tokenizer and filters to the index where it is rejected.
I'm afraid my understanding is not complete. Can you fill in the gaps?
Thanks,
Charles
----- Original Message -----
From: "Jack Krupansky" <ja...@gmail.com>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 4:31:22 PM
Subject: Re: Problem with solr.LengthFilterFactory
Sorry that my brain has turned to mush... the issue you are hitting is due
to a known, undocumented limit in the whitespace tokenizer:
https://issues.apache.org/jira/browse/LUCENE-5785
"White space tokenizer has undocumented limit of 256 characters per token"
If you look at the parsed query you will see that two query terms were
generated. This is because the whitespace tokenizer will simply split long
tokens every 256 characters. So, your filter will never see a long term.
There is a note on the Jira (evidently by me!) that you can use the pattern
tokenizer as a workaround. But... if your term is a string anyway, you
could just use the keyword tokenizer.
-- Jack Krupansky
On Fri, May 15, 2015 at 4:06 PM, Charles Sanders <cs...@redhat.com>
wrote:
> Shawn,
> Thanks a bunch for working with me on this.
>
> I have deleted all records from my index. Stopped solr. Made the schema
> changes as requested. Started solr. Then insert the one test record. Then
> search. Still see the same results. No portal_package is not the unique
> key, its uri. Which is a string field.
>
> <field name="portal_package" type="text_std" indexed="true" stored="true"
> multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </fieldType>
>
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> }
>
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 47,
> "params": {
> "spellcheck": "true",
> "enableElevation": "false",
> "df": "allText",
> "echoParams": "all",
> "spellcheck.maxCollations": "5",
> "spellcheck.dictionary": "andreasAutoComplete",
> "spellcheck.count": "5",
> "spellcheck.collate": "true",
> "spellcheck.onlyMorePopular": "true",
> "rows": "10",
> "indent": "true",
> "q":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "_": "1431719989047",
> "debug": "query",
> "wt": "json"
> }
> },
> "response": {
> "numFound": 1,
> "start": 0,
> "docs": [
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package": [
>
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> ],
> "_version_": 1501267024421060600,
> "timestamp": "2015-05-15T19:56:43.247Z",
> "language": "en"
> }
> ]
> },
> "debug": {
> "rawquerystring":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "querystring":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "parsedquery":
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "parsedquery_toString":
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "QParser": "LuceneQParser"
> }
> }
>
>
>
>
>
> ----- Original Message -----
>
> From: "Shawn Heisey" <ap...@elyograg.org>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 3:29:19 PM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> > Then I looked back at your fieldType definition and noticed that you
> > are only defining an index analyzer. Remove the 'type="index"' part of
> > the analyzer config so it happens at both index and query time,
> > reindex, then try again.
>
> The reindex may be very important here. I would actually completely
> delete your data directory and restart Solr before reindexing, to be
> sure you don't have old recordsfrom any previous reindexes.
>
> http://wiki.apache.org/solr/HowToReindex
>
> I think this next part is unlikely, but I'm going to ask it anyway: Is
> the portal_package field your schema uniqueKey? If it is, that might be
> an additional source of problems. Using a solr.Textfield for a
> uniqueKey field causes Solr to behave in unexpected ways.
>
> Thanks,
> Shawn
>
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/15/2015 2:06 PM, Charles Sanders wrote:
> I have deleted all records from my index. Stopped solr. Made the schema changes as requested. Started solr. Then insert the one test record. Then search. Still see the same results. No portal_package is not the unique key, its uri. Which is a string field.
>
> <field name="portal_package" type="text_std" indexed="true" stored="true" multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </fieldType>
You got rid of <analyzer> and it's closing tag entirely, that's not
going to work. This is what you need, and you'll need to reindex the
doc again:
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="300"/>
</analyzer>
</fieldType>
Here's a temporary paste that you can copy/paste:
http://apaste.info/NXn
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Jack Krupansky <ja...@gmail.com>.
Sorry that my brain has turned to mush... the issue you are hitting is due
to a known, undocumented limit in the whitespace tokenizer:
https://issues.apache.org/jira/browse/LUCENE-5785
"White space tokenizer has undocumented limit of 256 characters per token"
If you look at the parsed query you will see that two query terms were
generated. This is because the whitespace tokenizer will simply split long
tokens every 256 characters. So, your filter will never see a long term.
There is a note on the Jira (evidently by me!) that you can use the pattern
tokenizer as a workaround. But... if your term is a string anyway, you
could just use the keyword tokenizer.
-- Jack Krupansky
On Fri, May 15, 2015 at 4:06 PM, Charles Sanders <cs...@redhat.com>
wrote:
> Shawn,
> Thanks a bunch for working with me on this.
>
> I have deleted all records from my index. Stopped solr. Made the schema
> changes as requested. Started solr. Then insert the one test record. Then
> search. Still see the same results. No portal_package is not the unique
> key, its uri. Which is a string field.
>
> <field name="portal_package" type="text_std" indexed="true" stored="true"
> multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </fieldType>
>
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> }
>
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 47,
> "params": {
> "spellcheck": "true",
> "enableElevation": "false",
> "df": "allText",
> "echoParams": "all",
> "spellcheck.maxCollations": "5",
> "spellcheck.dictionary": "andreasAutoComplete",
> "spellcheck.count": "5",
> "spellcheck.collate": "true",
> "spellcheck.onlyMorePopular": "true",
> "rows": "10",
> "indent": "true",
> "q":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "_": "1431719989047",
> "debug": "query",
> "wt": "json"
> }
> },
> "response": {
> "numFound": 1,
> "start": 0,
> "docs": [
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package": [
>
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> ],
> "_version_": 1501267024421060600,
> "timestamp": "2015-05-15T19:56:43.247Z",
> "language": "en"
> }
> ]
> },
> "debug": {
> "rawquerystring":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "querystring":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "parsedquery":
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "parsedquery_toString":
> "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
> portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "QParser": "LuceneQParser"
> }
> }
>
>
>
>
>
> ----- Original Message -----
>
> From: "Shawn Heisey" <ap...@elyograg.org>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 3:29:19 PM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> > Then I looked back at your fieldType definition and noticed that you
> > are only defining an index analyzer. Remove the 'type="index"' part of
> > the analyzer config so it happens at both index and query time,
> > reindex, then try again.
>
> The reindex may be very important here. I would actually completely
> delete your data directory and restart Solr before reindexing, to be
> sure you don't have old recordsfrom any previous reindexes.
>
> http://wiki.apache.org/solr/HowToReindex
>
> I think this next part is unlikely, but I'm going to ask it anyway: Is
> the portal_package field your schema uniqueKey? If it is, that might be
> an additional source of problems. Using a solr.Textfield for a
> uniqueKey field causes Solr to behave in unexpected ways.
>
> Thanks,
> Shawn
>
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Shawn,
Thanks a bunch for working with me on this.
I have deleted all records from my index. Stopped solr. Made the schema changes as requested. Started solr. Then insert the one test record. Then search. Still see the same results. No portal_package is not the unique key, its uri. Which is a string field.
<field name="portal_package" type="text_std" indexed="true" stored="true" multiValued="true"/>
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="300" />
</fieldType>
{
"documentKind": "test",
"uri": "test300",
"id": "test300", "portal_package":"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
}
{
"responseHeader": {
"status": 0,
"QTime": 47,
"params": {
"spellcheck": "true",
"enableElevation": "false",
"df": "allText",
"echoParams": "all",
"spellcheck.maxCollations": "5",
"spellcheck.dictionary": "andreasAutoComplete",
"spellcheck.count": "5",
"spellcheck.collate": "true",
"spellcheck.onlyMorePopular": "true",
"rows": "10",
"indent": "true",
"q": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"_": "1431719989047",
"debug": "query",
"wt": "json"
}
},
"response": {
"numFound": 1,
"start": 0,
"docs": [
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": [
"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
],
"_version_": 1501267024421060600,
"timestamp": "2015-05-15T19:56:43.247Z",
"language": "en"
}
]
},
"debug": {
"rawquerystring": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"querystring": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"parsedquery": "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456 portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"parsedquery_toString": "portal_package:1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456 portal_package:7890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"QParser": "LuceneQParser"
}
}
----- Original Message -----
From: "Shawn Heisey" <ap...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 3:29:19 PM
Subject: Re: Problem with solr.LengthFilterFactory
On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> Then I looked back at your fieldType definition and noticed that you
> are only defining an index analyzer. Remove the 'type="index"' part of
> the analyzer config so it happens at both index and query time,
> reindex, then try again.
The reindex may be very important here. I would actually completely
delete your data directory and restart Solr before reindexing, to be
sure you don't have old recordsfrom any previous reindexes.
http://wiki.apache.org/solr/HowToReindex
I think this next part is unlikely, but I'm going to ask it anyway: Is
the portal_package field your schema uniqueKey? If it is, that might be
an additional source of problems. Using a solr.Textfield for a
uniqueKey field causes Solr to behave in unexpected ways.
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/15/2015 1:23 PM, Shawn Heisey wrote:
> Then I looked back at your fieldType definition and noticed that you
> are only defining an index analyzer. Remove the 'type="index"' part of
> the analyzer config so it happens at both index and query time,
> reindex, then try again.
The reindex may be very important here. I would actually completely
delete your data directory and restart Solr before reindexing, to be
sure you don't have old recordsfrom any previous reindexes.
http://wiki.apache.org/solr/HowToReindex
I think this next part is unlikely, but I'm going to ask it anyway: Is
the portal_package field your schema uniqueKey? If it is, that might be
an additional source of problems. Using a solr.Textfield for a
uniqueKey field causes Solr to behave in unexpected ways.
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/15/2015 12:47 PM, Charles Sanders wrote:
> Ran the same test as below. Added echoParams=all and debug=query. Thanks for the help!
Initially, I was a little confused by something I saw in your debug
output. I've broken up the query text with newlines into 50 character
increments for clarity in an email context:
"parsedquery": "portal_package:
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345
portal_package:67890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890
1234567890",
Then I looked back at your fieldType definition and noticed that you are
only defining an index analyzer. Remove the 'type="index"' part of the
analyzer config so it happens at both index and query time, reindex,
then try again.
If your index is small enough (especially if this is the only doc it
contains), you can go to the admin UI and load the terms with the Schema
Browser after selecting the portal_package field and see what is
actually in your index for this document.
As long as your analysis chain is the way it is, Solr's behavior will be
odd. Once we get it straightened out, then we can look for any other
problems.
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Ran the same test as below. Added echoParams=all and debug=query. Thanks for the help!
Results
{
"responseHeader": {
"status": 0,
"QTime": 46,
"params": {
"spellcheck": "true",
"enableElevation": "false",
"df": "allText",
"echoParams": "all",
"spellcheck.maxCollations": "5",
"spellcheck.dictionary": "andreasAutoComplete",
"spellcheck.count": "5",
"spellcheck.collate": "true",
"spellcheck.onlyMorePopular": "true",
"rows": "10",
"indent": "true",
"q": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"_": "1431715486263",
"debug": "query",
"wt": "json"
}
},
"response": {
"numFound": 1,
"start": 0,
"docs": [
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": [
"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
],
"_version_": 1501262378591846400,
"timestamp": "2015-05-15T18:42:52.625Z",
"language": "en"
}
]
},
"debug": {
"rawquerystring": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"querystring": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"parsedquery": "portal_package:123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345 portal_package:67890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"parsedquery_toString": "portal_package:123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345 portal_package:67890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"QParser": "LuceneQParser"
}
}
----- Original Message -----
From: "Shawn Heisey" <ap...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 2:36:03 PM
Subject: Re: Problem with solr.LengthFilterFactory
On 5/15/2015 10:04 AM, Charles Sanders wrote:
> Agree 100%. The value returned is the value stored. Not affected by the analyzer.
>
> However, I searched for that token. See my query? I would expect the analyzer to remove the large token. So that when I search for the large token I would find nothing. Rather it returns my record.
>
> Am I missing something here?
Can you add echoParams=all and debug=query to your request and send the
response you get?
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/15/2015 10:04 AM, Charles Sanders wrote:
> Agree 100%. The value returned is the value stored. Not affected by the analyzer.
>
> However, I searched for that token. See my query? I would expect the analyzer to remove the large token. So that when I search for the large token I would find nothing. Rather it returns my record.
>
> Am I missing something here?
Can you add echoParams=all and debug=query to your request and send the
response you get?
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Agree 100%. The value returned is the value stored. Not affected by the analyzer.
However, I searched for that token. See my query? I would expect the analyzer to remove the large token. So that when I search for the large token I would find nothing. Rather it returns my record.
Am I missing something here?
----- Original Message -----
From: "Jack Krupansky" <ja...@gmail.com>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 11:56:51 AM
Subject: Re: Problem with solr.LengthFilterFactory
The returned value is the stored or original source value - only the
indexed terms are affected by token filtering.
You could use an update processor if you want to adjust the actual source
value, such as the truncate processor to truncate long source values:
http://lucene.apache.org/solr/5_1_0/solr-core/org/apache/solr/update/processor/TruncateFieldUpdateProcessorFactory.html
-- Jack Krupansky
On Fri, May 15, 2015 at 11:38 AM, Charles Sanders <cs...@redhat.com>
wrote:
> Yes, that is what I am seeing. Looking in the code myself, I see no reason
> for this behavior. That is why I assumed I was doing something very wrong.
>
> Below I have included an example. I set the max length to 300. I insert a
> record with a single token of 500 characters. I expect the token to be
> removed and not included in the index. When I query using the large token,
> the record is returned. I can see the same result using the analysis page
> in the solr console.
>
> He is a test example:
>
> <field name="portal_package" type="text_std" indexed="true" stored="true"
> multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </analyzer>
> </fieldType>
>
>
> A test record:
>
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package":
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> }
>
>
> Query result:
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 55,
> "params": {
> "indent": "true",
> "q":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "_": "1431704135745",
> "wt": "json"
> }
> },
> "response": {
> "numFound": 1,
> "start": 0,
> "docs": [
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package": [
>
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> ],
> "_version_": 1501249997589446700,
> "timestamp": "2015-05-15T15:26:05.205Z",
> "language": "en"
> }
> ]
> }
> }
>
>
>
>
>
> ----- Original Message -----
>
> From: "Shawn Heisey" <ap...@elyograg.org>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 11:13:14 AM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> On 5/15/2015 8:49 AM, Charles Sanders wrote:
> > I'm seeing a problem with the LengthFilter. It appears to work fine
> until I increase the max value above 254. At the point it stops removing
> the very large token from the stream. As a result I get the error:
> > java.lang.IllegalArgumentException: Document contains at least one
> immense term...... UTF8 encoding is longer than the max length 32766
> >
> > I'm certain I'm doing this wrong. Can someone please show me the light.
> :)
> >
> > <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> > <analyzer type="index">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.LengthFilterFactory" min="1" max="254" />
> > </analyzer>
> > </fieldType>
>
> So with max="254", you don't get the error? Looking at the code for
> LengthFilter, I can't see any way for it to behave differently with a
> max of 254 vs. a max of 255 or higher. All of the interfaces and
> classes involved use "int" for length, which means it should work
> perfectly with numbers above 254.
>
> Thanks,
> Shawn
>
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Jack Krupansky <ja...@gmail.com>.
The returned value is the stored or original source value - only the
indexed terms are affected by token filtering.
You could use an update processor if you want to adjust the actual source
value, such as the truncate processor to truncate long source values:
http://lucene.apache.org/solr/5_1_0/solr-core/org/apache/solr/update/processor/TruncateFieldUpdateProcessorFactory.html
-- Jack Krupansky
On Fri, May 15, 2015 at 11:38 AM, Charles Sanders <cs...@redhat.com>
wrote:
> Yes, that is what I am seeing. Looking in the code myself, I see no reason
> for this behavior. That is why I assumed I was doing something very wrong.
>
> Below I have included an example. I set the max length to 300. I insert a
> record with a single token of 500 characters. I expect the token to be
> removed and not included in the index. When I query using the large token,
> the record is returned. I can see the same result using the analysis page
> in the solr console.
>
> He is a test example:
>
> <field name="portal_package" type="text_std" indexed="true" stored="true"
> multiValued="true"/>
>
> <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="300" />
> </analyzer>
> </fieldType>
>
>
> A test record:
>
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package":
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> }
>
>
> Query result:
>
> {
> "responseHeader": {
> "status": 0,
> "QTime": 55,
> "params": {
> "indent": "true",
> "q":
> "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
> "_": "1431704135745",
> "wt": "json"
> }
> },
> "response": {
> "numFound": 1,
> "start": 0,
> "docs": [
> {
> "documentKind": "test",
> "uri": "test300",
> "id": "test300",
> "portal_package": [
>
> "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
> ],
> "_version_": 1501249997589446700,
> "timestamp": "2015-05-15T15:26:05.205Z",
> "language": "en"
> }
> ]
> }
> }
>
>
>
>
>
> ----- Original Message -----
>
> From: "Shawn Heisey" <ap...@elyograg.org>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2015 11:13:14 AM
> Subject: Re: Problem with solr.LengthFilterFactory
>
> On 5/15/2015 8:49 AM, Charles Sanders wrote:
> > I'm seeing a problem with the LengthFilter. It appears to work fine
> until I increase the max value above 254. At the point it stops removing
> the very large token from the stream. As a result I get the error:
> > java.lang.IllegalArgumentException: Document contains at least one
> immense term...... UTF8 encoding is longer than the max length 32766
> >
> > I'm certain I'm doing this wrong. Can someone please show me the light.
> :)
> >
> > <fieldType name="text_std" class="solr.TextField"
> positionIncrementGap="100">
> > <analyzer type="index">
> > <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> > <filter class="solr.LengthFilterFactory" min="1" max="254" />
> > </analyzer>
> > </fieldType>
>
> So with max="254", you don't get the error? Looking at the code for
> LengthFilter, I can't see any way for it to behave differently with a
> max of 254 vs. a max of 255 or higher. All of the interfaces and
> classes involved use "int" for length, which means it should work
> perfectly with numbers above 254.
>
> Thanks,
> Shawn
>
>
>
Re: Problem with solr.LengthFilterFactory
Posted by Charles Sanders <cs...@redhat.com>.
Yes, that is what I am seeing. Looking in the code myself, I see no reason for this behavior. That is why I assumed I was doing something very wrong.
Below I have included an example. I set the max length to 300. I insert a record with a single token of 500 characters. I expect the token to be removed and not included in the index. When I query using the large token, the record is returned. I can see the same result using the analysis page in the solr console.
He is a test example:
<field name="portal_package" type="text_std" indexed="true" stored="true" multiValued="true"/>
<fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LengthFilterFactory" min="1" max="300" />
</analyzer>
</fieldType>
A test record:
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": "12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
}
Query result:
{
"responseHeader": {
"status": 0,
"QTime": 55,
"params": {
"indent": "true",
"q": "portal_package:12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890",
"_": "1431704135745",
"wt": "json"
}
},
"response": {
"numFound": 1,
"start": 0,
"docs": [
{
"documentKind": "test",
"uri": "test300",
"id": "test300",
"portal_package": [
"12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"
],
"_version_": 1501249997589446700,
"timestamp": "2015-05-15T15:26:05.205Z",
"language": "en"
}
]
}
}
----- Original Message -----
From: "Shawn Heisey" <ap...@elyograg.org>
To: solr-user@lucene.apache.org
Sent: Friday, May 15, 2015 11:13:14 AM
Subject: Re: Problem with solr.LengthFilterFactory
On 5/15/2015 8:49 AM, Charles Sanders wrote:
> I'm seeing a problem with the LengthFilter. It appears to work fine until I increase the max value above 254. At the point it stops removing the very large token from the stream. As a result I get the error:
> java.lang.IllegalArgumentException: Document contains at least one immense term...... UTF8 encoding is longer than the max length 32766
>
> I'm certain I'm doing this wrong. Can someone please show me the light. :)
>
> <fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="254" />
> </analyzer>
> </fieldType>
So with max="254", you don't get the error? Looking at the code for
LengthFilter, I can't see any way for it to behave differently with a
max of 254 vs. a max of 255 or higher. All of the interfaces and
classes involved use "int" for length, which means it should work
perfectly with numbers above 254.
Thanks,
Shawn
Re: Problem with solr.LengthFilterFactory
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/15/2015 8:49 AM, Charles Sanders wrote:
> I'm seeing a problem with the LengthFilter. It appears to work fine until I increase the max value above 254. At the point it stops removing the very large token from the stream. As a result I get the error:
> java.lang.IllegalArgumentException: Document contains at least one immense term...... UTF8 encoding is longer than the max length 32766
>
> I'm certain I'm doing this wrong. Can someone please show me the light. :)
>
> <fieldType name="text_std" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <filter class="solr.LengthFilterFactory" min="1" max="254" />
> </analyzer>
> </fieldType>
So with max="254", you don't get the error? Looking at the code for
LengthFilter, I can't see any way for it to behave differently with a
max of 254 vs. a max of 255 or higher. All of the interfaces and
classes involved use "int" for length, which means it should work
perfectly with numbers above 254.
Thanks,
Shawn