You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "vvivekiyer (via GitHub)" <gi...@apache.org> on 2023/07/19 05:22:48 UTC
[GitHub] [pinot] vvivekiyer opened a new issue, #11130: Bug in reload segment when indexes are removed
vvivekiyer opened a new issue, #11130:
URL: https://github.com/apache/pinot/issues/11130
We noticed that index removal threw an exception during segment reload in certain cases. This error is not deterministic and only occurs for some segments.
**Stack-trace**
> 2023/07/17 23:33:53.329 INFO [InvertedIndexHandler] [HelixTaskExecutor-message_handle_thread_25] [pinot-server] [] Creating new inverted index for segment: <segmentName> column: <columnName>
> 2023/07/17 23:34:23.516 INFO [InvertedIndexHandler] [HelixTaskExecutor-message_handle_thread_25] [pinot-server] [] Created inverted index for segment: <segmentName>, column: <columnName>
> 2023/07/17 23:34:23.516 INFO [BloomFilterHandler] [HelixTaskExecutor-message_handle_thread_25] [pinot-server] [] Removing existing bloom filter from segment: <segmentName> column: <columnName>
> 2023/07/17 23:34:23.516 INFO [BloomFilterHandler] [HelixTaskExecutor-message_handle_thread_25] [pinot-server] [] Removed existing bloom filter from segment: <segmentName>, column: <columnName>
> 2023/07/17 23:34:28.344 ERROR [PinotDataBuffer] [HelixTaskExecutor-message_handle_thread_25] [pinot-server] [] Caught exception while mapping file: segmentDir/segment_2023-04-28_2023-04-28_0/v3/columns.psf from offset: 2480743340 of size: 4049668988 with description: SingleFileIndexDirectory.segmentDir/segment_2023-04-28_2023-04-28_0/v3/columns.psf.single_file_index.rw..2480743340.4049668988
>
>
>
> java.io.IOException: Bad file descriptor
> at java.io.RandomAccessFile.write0(Native Method) ~[?:?]
> at java.io.RandomAccessFile.write(RandomAccessFile.java:523) ~[?:?]
> at xerial.larray.mmap.MMapBuffer.<init>(MMapBuffer.java:87) ~[larray-mmap-0.4.1.jar:0.4.1]
> at org.apache.pinot.segment.spi.memory.PinotNonNativeOrderLBuffer.mapFile(PinotNonNativeOrderLBuffer.java:49) ~[pinot-segment-spi-0.13.0-dev-852.jar:0.13.0-dev-852-0e88f54510e58d61ce00337ae01a860368683a73]
> at org.apache.pinot.segment.spi.memory.PinotDataBuffer.mapFile(PinotDataBuffer.java:196) [pinot-segment-spi-0.13.0-dev-852.jar:0.13.0-dev-852-0e88f54510e58d61ce00337ae01a860368683a73]
> at org.apache.pinot.segment.local.segment.store.SingleFileIndexDirectory.mapAndSliceFile(SingleFileIndexDirectory.java:293) [pinot-segment-local-0.13.0-dev-852.jar:0.13.0-dev-852-0e88f54510e58d61ce00337ae01a860368683a73]
> at org.apache.pinot.segment.local.segment.store.SingleFileIndexDirectory.mapBufferEntries(SingleFileIndexDirectory.java:264) [pinot-segment-local-0.13.0-dev-852.jar:0.13.0-dev-852-0e88f54510e58d61ce00337ae01a860368683a73]
Debugging this, it looks like the columns.psf file has incomplete data.
> ❯ ls -lirt
> total 11075616
> 112226639 -rw-r--r-- 1 vvaidyan 101 16 Jul 18 17:50 creation.meta
> 112226641 -rw-r--r-- 1 vvaidyan 101 4779 Jul 18 17:50 metadata.properties
> 112226689 -rw-r--r-- 1 vvaidyan 101 5666876107 Jul 18 17:51 columns.psf
> 112226700 -rw-r--r-- 1 vvaidyan 101 1301 Jul 18 17:51 index_map
But index_map file refers to offset > size of columns.psf
> col1.dictionary.startOffset = 0
> col1.dictionary.size = 728547872
> col1.forward_index.startOffset = 728547872
> col1.forward_index.size = 442462399
> col1.inverted_index.startOffset = 1171010271
> col1.inverted_index.size = 1309733069
> col2.dictionary.startOffset = 2480743340
> col2.dictionary.size = 4049668988
> col2.forward_index.startOffset = 6530412328
Debugging this, it looks like our call to https://github.com/apache/pinot/blob/8fff7eb70597eb74c05ee788378eb8fa1ed8efaf/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java#L420 is incorrect. [FileChannel.transferTo](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#transferTo-long-long-java.nio.channels.WritableByteChannel-) is not guaranteed to transfer the entire data and hence should be called within a loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #11130: Bug in reload segment when indexes are removed
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #11130: Bug in reload segment when indexes are removed
URL: https://github.com/apache/pinot/issues/11130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org