You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/03/03 12:52:47 UTC
[GitHub] [pinot] richardstartin opened a new pull request #8286: add more testing for MV var-length columns, detect overflow scenarios
richardstartin opened a new pull request #8286:
URL: https://github.com/apache/pinot/pull/8286
This addresses a bug report of a buffer overflow creating MV string raw forward index, with the following stack trace:
```
Start building IndexCreator!
Could not build segment
java.lang.IllegalArgumentException: newPosition > limit: (1048580 > 1048575)
at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
at java.nio.ByteBuffer.position(ByteBuffer.java:1094) ~[?:?]
at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226) ~[?:?]
at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67) ~[?:?]
at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putStrings(VarByteChunkSVForwardIndexWriter.java:119) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator.putStringMV(MultiValueVarByteRawIndexCreator.java:106) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:543) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:815) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:746) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:644) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
I have added extensive testing for the impacted components and could not find any buffer overflows, and have narrowed this down to an integer overflow such that the product of `Integer.BYTES + maxRowLengthInBytes + maxLengthPrefixes` and a fixed `numDocsPerChunk` overflows and is non-negative. Very large MV values is a limitation of the current implementation and I have added precondition checks to detect overflow unambiguously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin merged pull request #8286: add more testing for MV var-length columns, detect overflow scenarios
Posted by GitBox <gi...@apache.org>.
richardstartin merged pull request #8286:
URL: https://github.com/apache/pinot/pull/8286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org