You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Claus Stadler (Jira)" <ji...@apache.org> on 2021/01/31 03:54:00 UTC
[jira] [Commented] (HADOOP-17453) BZip2Codec incorrectly throws
IndexOutOfBoundsException: offs(X) + len(X+1) > dest.length(Y).
[ https://issues.apache.org/jira/browse/HADOOP-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275771#comment-17275771 ]
Claus Stadler commented on HADOOP-17453:
----------------------------------------
I am facing IndexOutOfBounds exceptions with my custom RecordReader on hadoop common 2.8.5 and I am also inclined to think this is a major bug in this line.
If I understand this codec stuff correctly (not claiming I do) reading decoded data (e.g. text) with READ_MODE.BY_BLOCK mode backed by an encoded stream (e.g. bzip2) should make a read on the decoded stream return when the backing stream hits its set boundary (typically the split end); and this mechanism is referred to as "advertise".
But before my repeated reads actually hit the split boundary I get an IndexOutOfBounds exception - apparently because my buffer's length is at some point less than 2 * offset + 1 - huh?
> BZip2Codec incorrectly throws IndexOutOfBoundsException: offs(X) + len(X+1) > dest.length(Y).
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-17453
> URL: https://issues.apache.org/jira/browse/HADOOP-17453
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.1.2
> Reporter: Christian Asmussen
> Priority: Major
>
> In org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream
> around line 496 seems to mistakenly add the offset to the length.
> {noformat}
> if (this.posSM == POS_ADVERTISEMENT_STATE_MACHINE.ADVERTISE) {
> result = this.input.read(b, off, off + 1 << HERE);
> {noformat}
> Here's a reference [BZip2Codec.java:L496|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/BZip2Codec.java#L496]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org