You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/12/17 20:58:58 UTC
[GitHub] [pinot] mikecdavis-dd opened a new issue #7923: IndexOutOfBoundsException when generating REALTIME inverted index
mikecdavis-dd opened a new issue #7923:
URL: https://github.com/apache/pinot/issues/7923
After migrating from `0.8.0` to `0.9.2` we started to see the receive the following exception for some of our high volume REALTIME tables:
```
java.lang.IndexOutOfBoundsException: Index 317 out of bounds for length 151
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:372) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeInvertedIndexReader.add(RealtimeInvertedIndexReader.java:59) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:580) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:491) ~[pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:524) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.2-jar-with-dependencies.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
Another similar example:
```
java.lang.IndexOutOfBoundsException: Index 3021 out of bounds for length 27
```
Digging into the stack trace a bit this [method](https://github.com/apache/pinot/blob/f4e55182a8c552c8493d1693868b0c752b1a1d56/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java#L47-L61) appears to operate on the assumption that the `dictId` is either the next index in the underlying `ArrayList` and should be added, or it's already a valid index and could be used to `get(dictId)`.
When this assumption is not met then an `IndexOutOfBoundsException` is thrown and the table stops consuming from that partition.
Note: No table configurations were made between the version upgrades.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7923: IndexOutOfBoundsException when generating REALTIME inverted index
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7923:
URL: https://github.com/apache/pinot/issues/7923#issuecomment-997097065
Can you please check if there are other errors of `Caught exception while transforming the record: ...` which is not caused by `IndexOutOfBoundsException`? Based on the code, I think the `IndexOutOfBoundsException` can happen when some of the previous records are not indexed properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #7923: IndexOutOfBoundsException when generating REALTIME inverted index
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #7923:
URL: https://github.com/apache/pinot/issues/7923#issuecomment-1042404990
@richardstartin This is related to my comment about `RealtimeInvertedIndexReader` under your PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on issue #7923: IndexOutOfBoundsException when generating REALTIME inverted index
Posted by GitBox <gi...@apache.org>.
richardstartin commented on issue #7923:
URL: https://github.com/apache/pinot/issues/7923#issuecomment-1042393289
I haven't reproduced this stack trace but I think this may be fixed by #8211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org