You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by Mats Norén <ma...@gmail.com> on 2007/12/19 20:43:18 UTC
Problem with WordParser
Hello,
I've been trying to extract text from a couple of different MS-Word
files and I'm getting mixed results.
Almost by random (as I see it) I get this error:
java.lang.StringIndexOutOfBoundsException: String index out of range: -21047
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:886)
at java.lang.StringBuffer.substring(StringBuffer.java:417)
at org.apache.poi.hwpf.model.TextPiece.substring(TextPiece.java:88)
at org.apache.tika.parser.microsoft.WordParser.extractText(WordParser.java:163)
Looking at the TextPiece in POI I can see that the substring method is
called with a negative value for end
public String substring(int start, int end)
{
int denominator = _usesUnicode ? 2 : 1;
return ((StringBuffer)_buf).substring(start/denominator, end/denominator);
}
I just can't see why / how runEnd - currentTextStart can end up being
a negative value.
String str = currentPiece.substring(0, runEnd - currentTextStart);
Any ideas?
Regards Mats
Re: Problem with WordParser
Posted by Niall Pemberton <ni...@gmail.com>.
On Dec 19, 2007 7:43 PM, Mats Norén <ma...@gmail.com> wrote:
> Hello,
> I've been trying to extract text from a couple of different MS-Word
> files and I'm getting mixed results.
> Almost by random (as I see it) I get this error:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -21047
> at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:886)
> at java.lang.StringBuffer.substring(StringBuffer.java:417)
> at org.apache.poi.hwpf.model.TextPiece.substring(TextPiece.java:88)
> at org.apache.tika.parser.microsoft.WordParser.extractText(WordParser.java:163)
>
> Looking at the TextPiece in POI I can see that the substring method is
> called with a negative value for end
>
> public String substring(int start, int end)
> {
> int denominator = _usesUnicode ? 2 : 1;
>
> return ((StringBuffer)_buf).substring(start/denominator, end/denominator);
> }
>
> I just can't see why / how runEnd - currentTextStart can end up being
> a negative value.
>From my reading of the code I can't see how it can be anything other
than zero or negative if/when it gets to line 163 of Tika's WordParser
- since before that it loops until runEnd is less than or equal to
currentTextEnd:
while (runEnd > currentTextEnd) {
...
}
String str = currentPiece.substring(0, runEnd - currentTextStart);
IMO this is a Tika bug and you should file a bug report (preferrably
with an attached example Word document that causes the issue):
https://issues.apache.org/jira/browse/TIKA
Niall
> String str = currentPiece.substring(0, runEnd - currentTextStart);
>
> Any ideas?
>
> Regards Mats
>
Re: Problem with WordParser
Posted by Mats Norén <ma...@gmail.com>.
Done.
It seems that the error occurs on the last TextPiece if it occurs at all..
On Dec 20, 2007 8:11 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Dec 20, 2007 1:54 PM, Mats Norén <ma...@gmail.com> wrote:
> > On Dec 20, 2007 11:03 AM, Jukka Zitting <ju...@gmail.com> wrote:
> > > You may want to contact the POI mailing lists, as I don't think many
> > > of us have too much experience with POI internals.
> >
> > The thing is that it's the WordParser.java that calls
> > TextPiece.substring with a negative value, so my guess is that it's
> > the algorithm itself that for some corner case does the wrong thing.
>
> I'm sorry, you're of course right. I somehow misunderstood you
> referring to a class within POI. Too much on my mind lately...
>
> As Niall already pointed out, this seems like a bug in our parser code.
>
> > It's my understaning that the text extraction in Jackrabbit is based
> > on textmining.org which is the basis for the WordParser in Tika, is
> > that correct or have got it wrong?
>
> Yes. The code actually ended in Tika through Nutch and Lius, but
> originates from the same textmining.org codebase that also Jackrabbit
> is currently using. Unfortunately Ryan Ackley is no longer maintaining
> the code, which leaves us with few options other than embedding the
> code in Tika. It would IMHO be best if we could push all the complex
> file format logic out to separate parser libraries (preferably POI in
> this case), where there would likely be people with much better
> understanding of that specific format and the related parsing code.
> Anyway, until we get rid of the code we should try to maintain it the
> best we can, so please file a bug report for this issue. :-)
>
> BR,
>
> Jukka Zitting
>
Re: Problem with WordParser
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Dec 20, 2007 1:54 PM, Mats Norén <ma...@gmail.com> wrote:
> On Dec 20, 2007 11:03 AM, Jukka Zitting <ju...@gmail.com> wrote:
> > You may want to contact the POI mailing lists, as I don't think many
> > of us have too much experience with POI internals.
>
> The thing is that it's the WordParser.java that calls
> TextPiece.substring with a negative value, so my guess is that it's
> the algorithm itself that for some corner case does the wrong thing.
I'm sorry, you're of course right. I somehow misunderstood you
referring to a class within POI. Too much on my mind lately...
As Niall already pointed out, this seems like a bug in our parser code.
> It's my understaning that the text extraction in Jackrabbit is based
> on textmining.org which is the basis for the WordParser in Tika, is
> that correct or have got it wrong?
Yes. The code actually ended in Tika through Nutch and Lius, but
originates from the same textmining.org codebase that also Jackrabbit
is currently using. Unfortunately Ryan Ackley is no longer maintaining
the code, which leaves us with few options other than embedding the
code in Tika. It would IMHO be best if we could push all the complex
file format logic out to separate parser libraries (preferably POI in
this case), where there would likely be people with much better
understanding of that specific format and the related parsing code.
Anyway, until we get rid of the code we should try to maintain it the
best we can, so please file a bug report for this issue. :-)
BR,
Jukka Zitting
Re: Problem with WordParser
Posted by Mats Norén <ma...@gmail.com>.
On Dec 20, 2007 11:03 AM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> You may want to contact the POI mailing lists, as I don't think many
> of us have too much experience with POI internals.
The thing is that it's the WordParser.java that calls
TextPiece.substring with a negative value, so my guess is that it's
the algorithm itself that for some corner case does the wrong thing.
It's my understaning that the text extraction in Jackrabbit is based
on textmining.org which is the basis for the WordParser in Tika, is
that correct or have got it wrong?
/Mats
>
> BR,
>
> Jukka Zitting
>
Re: Problem with WordParser
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Dec 20, 2007 11:44 AM, Mats Norén <ma...@gmail.com> wrote:
> Any ideas?
You may want to contact the POI mailing lists, as I don't think many
of us have too much experience with POI internals.
BR,
Jukka Zitting
Fwd: Problem with WordParser
Posted by Mats Norén <ma...@gmail.com>.
Hello,
I've been trying to extract text from a couple of different MS-Word
files and I'm getting mixed results.
Almost by random (as I see it) I get this error:
java.lang.StringIndexOutOfBoundsException: String index out of range: -21047
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:886)
at java.lang.StringBuffer.substring(StringBuffer.java:417)
at org.apache.poi.hwpf.model.TextPiece.substring(TextPiece.java:88)
at org.apache.tika.parser.microsoft.WordParser.extractText(WordParser.java:163)
Looking at the TextPiece in POI I can see that the substring method is
called with a negative value for end
public String substring(int start, int end)
{
int denominator = _usesUnicode ? 2 : 1;
return ((StringBuffer)_buf).substring(start/denominator, end/denominator);
}
I just can't see why / how runEnd - currentTextStart can end up being
a negative value.
String str = currentPiece.substring(0, runEnd - currentTextStart);
Any ideas?
Regards Mats