You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "martijnvg (via GitHub)" <gi...@apache.org> on 2025/07/01 07:24:51 UTC
[I] Should SortingStoredFieldsConsumer do a full integrity check? [lucene]
martijnvg opened a new issue, #14881:
URL: https://github.com/apache/lucene/issues/14881
### Description
Today when index sorting is enabled and stored fields get flushed then the `SortingStoredFieldsConsumer` gets used in order to store stored fields in the order in which index sorting is configured. This class writes temp files to disk that then get read completely twice. The first time is to do an integrity check and the second time the temp files are read in random order. This to write stored fields in the right order (defined by index sorting) in the new segment.
During heavy indexing the fact that the stored field temp files are read twice is expensive. Especially given that these temp files will be removed after flushing has completed. In other formats (postings, bkd tree, quantized vectors), tmp files that get created during writing seem to be read only once. During reading either integrity is check using `Directory#openChecksumInput()` (only possible if temp file is read from beginning to end) or there is a footer check (reads and validates CRC, footer magic and algorithm id).
I wonder whether it makes sense to remove the full separate [integrity check](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/SortingStoredFieldsConsumer.java#L106) in `SortingStoredFieldsConsumer`? This can be quite costly, especially the integrity check for the temp fdt file and also there is already some light integrity checking via `CodecUtil.checkFooter(...)` in `Lucene90CompressingStoredFieldsReader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Should SortingStoredFieldsConsumer do a full integrity check? [lucene]
Posted by "martijnvg (via GitHub)" <gi...@apache.org>.
martijnvg commented on issue #14881:
URL: https://github.com/apache/lucene/issues/14881#issuecomment-3022712602
> we must fully check on merge:
I fully agree that on merge a full check is required.
I'm wondering whether we can avoid doing a full check at flush time for the tmp files that get created for stored fields when index sorting is enabled. From my understanding `SortingStoredFieldsConsumer` is only used when flushing a new segment. At merge time, there is no need to create a tmp file for stored fields. I noticed that in other formats, when creating a tmp file not always a full check is performed. With that in mind, I wonder whether we can omit the full integrity check that `SortingStoredFieldsConsumer` is doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Should SortingStoredFieldsConsumer do a full integrity check? [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #14881:
URL: https://github.com/apache/lucene/issues/14881#issuecomment-3022484768
we must fully check on merge: there's no shortcutting this, otherwise corruption can spread via merges which is impossible to debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Should SortingStoredFieldsConsumer do a full integrity check? [lucene]
Posted by "martijnvg (via GitHub)" <gi...@apache.org>.
martijnvg closed issue #14881: Should SortingStoredFieldsConsumer do a full integrity check?
URL: https://github.com/apache/lucene/issues/14881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org