You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Lu <ch...@gmail.com> on 2010/01/14 22:38:45 UTC

IllegalArgumentException when IndexWriter.addDocument

Seems a integer overflow problem?

java.lang.IllegalArgumentException: Increment must be zero or greater: 
-472893952
  at 
org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl.setPositionIncrement(PositionIncrementAttributeImpl.java:58)
  at 
org.apache.lucene.analysis.StopFilter.incrementToken(StopFilter.java:256)
  at 
org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:188)
  at 
org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
  at 
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
  at 
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1931)
  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1905)


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: IllegalArgumentException when IndexWriter.addDocument

Posted by Simon Willnauer <si...@googlemail.com>.
Which analyzer are you using?

simon
On Thu, Jan 14, 2010 at 10:40 PM, Chris Lu <ch...@gmail.com> wrote:
> Notes: I am using Lucene 3.0
>>
>> Seems a integer overflow problem?
>>
>> java.lang.IllegalArgumentException: Increment must be zero or greater:
>> -472893952
>>  at
>> org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl.setPositionIncrement(PositionIncrementAttributeImpl.java:58)
>>  at
>> org.apache.lucene.analysis.StopFilter.incrementToken(StopFilter.java:256)
>>  at
>> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:188)
>>  at
>> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)
>>  at
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)
>>  at
>> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)
>>  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1931)
>>  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1905)
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: IllegalArgumentException when IndexWriter.addDocument

Posted by Chris Lu <ch...@gmail.com>.
I am using a custom analyzer upgrading from Lucene 2.x.
Need to get more familiar with Lucene 3.0 behavior.
I think this is one of the upgrade pitfalls.

Thanks for the help!

Chris

Uwe Schindler wrote:
> This problem occurs, if you have a Tokenizer or TokenFilter that produces new tokens but does not call clearAttributes(). What TokenStreams do you use in your analyzer?
>
> If you not call clearAttributes() (see javadocs of Tokenizer!) whenever you produce new tokens (in any type of TokenStream), the positionIncrement is never reset back to 1 (and when in adiition the analyzer reuses the tokenstream, it is really never reset even across multiple documents). So whenever a stopword occurs it get larger...
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>   
>> -----Original Message-----
>> From: Chris Lu [mailto:chris.lu@gmail.com]
>> Sent: Thursday, January 14, 2010 10:41 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: IllegalArgumentException when IndexWriter.addDocument
>>
>> Notes: I am using Lucene 3.0
>>     
>>> Seems a integer overflow problem?
>>>
>>> java.lang.IllegalArgumentException: Increment must be zero or
>>>       
>> greater:
>>     
>>> -472893952
>>>  at
>>>
>>>       
>> org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeIm
>> pl.setPositionIncrement(PositionIncrementAttributeImpl.java:58)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.analysis.StopFilter.incrementToken(StopFilter.java:25
>> 6)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPe
>> rField.java:188)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocF
>> ieldProcessorPerThread.java:246)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.
>> java:774)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.jav
>> a:752)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1931)
>>     
>>>  at
>>>
>>>       
>> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1905)
>>     
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>     
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: IllegalArgumentException when IndexWriter.addDocument

Posted by Uwe Schindler <uw...@thetaphi.de>.
This problem occurs, if you have a Tokenizer or TokenFilter that produces new tokens but does not call clearAttributes(). What TokenStreams do you use in your analyzer?

If you not call clearAttributes() (see javadocs of Tokenizer!) whenever you produce new tokens (in any type of TokenStream), the positionIncrement is never reset back to 1 (and when in adiition the analyzer reuses the tokenstream, it is really never reset even across multiple documents). So whenever a stopword occurs it get larger...

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Chris Lu [mailto:chris.lu@gmail.com]
> Sent: Thursday, January 14, 2010 10:41 PM
> To: java-user@lucene.apache.org
> Subject: Re: IllegalArgumentException when IndexWriter.addDocument
> 
> Notes: I am using Lucene 3.0
> > Seems a integer overflow problem?
> >
> > java.lang.IllegalArgumentException: Increment must be zero or
> greater:
> > -472893952
> >  at
> >
> org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeIm
> pl.setPositionIncrement(PositionIncrementAttributeImpl.java:58)
> >
> >  at
> >
> org.apache.lucene.analysis.StopFilter.incrementToken(StopFilter.java:25
> 6)
> >  at
> >
> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPe
> rField.java:188)
> >
> >  at
> >
> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocF
> ieldProcessorPerThread.java:246)
> >
> >  at
> >
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.
> java:774)
> >
> >  at
> >
> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.jav
> a:752)
> >
> >  at
> >
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1931)
> >  at
> >
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1905)
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: IllegalArgumentException when IndexWriter.addDocument

Posted by Chris Lu <ch...@gmail.com>.
Notes: I am using Lucene 3.0
> Seems a integer overflow problem?
>
> java.lang.IllegalArgumentException: Increment must be zero or greater: 
> -472893952
>  at 
> org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl.setPositionIncrement(PositionIncrementAttributeImpl.java:58) 
>
>  at 
> org.apache.lucene.analysis.StopFilter.incrementToken(StopFilter.java:256)
>  at 
> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:188) 
>
>  at 
> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246) 
>
>  at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774) 
>
>  at 
> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752) 
>
>  at 
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1931)
>  at 
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1905)
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org