You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jerry-024 (via GitHub)" <gi...@apache.org> on 2026/09/10 02:31:42 UTC

[PR] feat: optimize IVF index build reads [paimon-rust]

jerry-024 opened a new pull request, #800:
URL: https://github.com/apache/paimon-rust/pull/800

   <!--
   Thank you very much for contributing to Paimon Rust - we are happy that you want to help us improve it. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/paimon-rust/issues). Exceptions are made for typos in documentation or comments, which need no issue.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
   
     - Make sure that the change passes the automated tests, i.e., `cargo test` passes.
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
   
   **(The sections below can be removed for hotfixes or typos)**
   -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #xxx
   
   <!-- What is the purpose of the change -->
   
   ### Brief change log
   
   <!-- Please describe the changes made in this pull request and explain how they address the issue -->
   
   ### Tests
   
   <!-- List unit tests or integration cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature or require documentation updates -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #800:
URL: https://github.com/apache/paimon-rust/pull/800#discussion_r3986220832


##########
crates/paimon/src/table/vindex_index_build_builder/writer.rs:
##########
@@ -87,245 +93,571 @@ impl<'a> VindexIndexBuildBuilder<'a> {
         let expected_bytes = checked_vector_bytes(row_count_usize, dimension_usize)?;
         let training_vector_count =
             checked_training_vector_count(row_count_usize, options.train_sample_ratio)?;
-        let training_buffer_rows =
-            (VECTOR_BUFFER_BYTES / checked_vector_bytes(1, dimension_usize)?).max(1);
-        let training_buffer_floats = training_buffer_rows
-            .checked_mul(dimension_usize)
-            .ok_or_else(|| Error::DataInvalid {
-                message: "vindex training buffer length overflows usize".to_string(),
-                source: None,
-            })?;
+        let training_rows_retained = if self.index_type == DISKANN_IDENTIFIER {
+            0
+        } else {
+            default_training_vector_count(training_vector_count, options.config.nlist()).map_err(
+                |e| Error::DataInvalid {
+                    message: format!("Failed to calculate IVF training vector count: {e}"),
+                    source: Some(Box::new(e)),
+                },
+            )?
+        };
+        let mut sparse_ranges =
+            if training_rows_retained > 0 && training_rows_retained < row_count_usize {
+                Some(plan_ivf_training_ranges(shard, training_rows_retained)?)
+            } else {
+                None
+            };
+        let capability_start = (timing_enabled && sparse_ranges.is_some()).then(Instant::now);
+        if let Some(ranges) = sparse_ranges.as_ref() {
+            let mut checks = Vec::new();
+            let mut usable = true;
+            for file in &shard.files {
+                let path = file.data_file_path(&shard.bucket_path);
+                if !path.to_ascii_lowercase().ends_with(".parquet") {
+                    usable = false;
+                    break;
+                }
+                let Some((file_start, file_end)) = file.row_id_range() else {
+                    usable = false;
+                    break;
+                };
+                let local_ranges = ranges
+                    .iter()
+                    .filter_map(|range| {
+                        let from = range.from().max(file_start);
+                        let to = range.to().min(file_end);
+                        (from <= to).then(|| RowRange::new(from - file_start, to - file_start))
+                    })
+                    .collect::<Vec<_>>();
+                if local_ranges.is_empty()
+                    || file
+                        .write_cols
+                        .as_ref()
+                        .is_some_and(|columns| !columns.iter().any(|column| column == index_column))
+                {
+                    continue;
+                }
+                let file_size = u64::try_from(file.file_size).map_err(|e| Error::DataInvalid {
+                    message: format!(
+                        "Invalid data file size for '{}': {}",
+                        file.file_name, file.file_size
+                    ),
+                    source: Some(Box::new(e)),
+                })?;
+                checks.push((path, file_size, local_ranges));
+            }
+            let found_vector_file = !checks.is_empty();
+            if usable && found_vector_file {
+                let concurrency = self
+                    .table
+                    .schema()
+                    .core_options()
+                    .parquet_row_group_parallelism()?;
+                let file_io = self.table.file_io();
+                let mut checks = futures::stream::iter(checks)
+                    .map(|(path, file_size, local_ranges)| async move {
+                        let input = file_io.new_input(&path)?;
+                        has_usable_offset_index(
+                            Box::new(input.reader().await?),
+                            file_size,
+                            index_column,
+                            &local_ranges,
+                        )
+                        .await

Review Comment:
   Rechecked at 28d6670: this remains reproducible with the current 512 short, jittered ranges. For the same default 100,000-row shard with 128-dimensional vectors, the sample and full scan still each fetch exactly 47,788,791 data bytes, excluding metadata. The latest concurrency and selected-page budget changes do not prevent the extra effectively full source pass. The sparse-path gate at `writer.rs:173` still only checks offset-index availability; please include actual page savings after range coalescing, or fall back when the sample reads essentially the entire projection.



##########
crates/paimon/src/table/vindex_index_build_builder/planning.rs:
##########
@@ -45,3 +51,78 @@ pub(super) fn plan_vindex_shards(
         "vindex",
     )
 }
+
+pub(super) fn plan_ivf_training_ranges(
+    shard: &VindexIndexShard,
+    training_rows: usize,
+) -> Result<Option<Vec<RowRange>>> {
+    let shard_rows = usize::try_from(checked_row_count(
+        shard.row_range_start,
+        shard.row_range_end,
+    )?)
+    .map_err(|error| Error::DataInvalid {
+        message: "vindex shard row count does not fit usize".to_string(),
+        source: Some(Box::new(error)),
+    })?;
+    if training_rows == 0 || training_rows > shard_rows {
+        return Err(Error::DataInvalid {
+            message: format!(
+                "Invalid IVF training row count: {training_rows}; shard contains {shard_rows} rows"
+            ),
+            source: None,
+        });
+    }
+    let range_count = training_rows.div_ceil(MAX_IVF_TRAINING_RANGE_ROWS);

Review Comment:
   [P2] Keep small training samples distributed across the shard
   
   When `training_rows <= 128`, this calculation produces just one contiguous range. A realistic incremental shard with 1,000 new rows and `train.sample-ratio=0.1` therefore trains on 100 adjacent rows, whereas the previous implementation sampled every tenth row across the whole shard.
   
   Using the actual planner and vindex 0.4 trainer, I reproduced this with 450 vectors `[0]`, 450 `[1]`, and 100 `[100]`, IVF-SQ, L2, and `nlist=nprobe=1`. For snapshot 1, bucket 0, and an empty partition, the new range is `[385,484]`, which excludes the entire final distribution. Querying `[100]` returns 10/10 results from the final cluster with the baseline, but 0/10 with this plan: SQ clamps those vectors to the older upper bound and returns rows 450–459. Please retain multiple strata for small samples, or fall back to the original sampling path, and cover this incremental-shard case in a recall regression test.



##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -642,13 +729,50 @@ impl FormatFileReader for ParquetFormatReader {
     }
 }
 
-fn projected_row_group_bytes(row_group: &RowGroupMetaData, projection: &ProjectionMask) -> u64 {
+fn projected_row_group_bytes(
+    row_group: &RowGroupMetaData,
+    projection: &ProjectionMask,
+    selection: Option<&RowSelection>,
+    offset_index: Option<&[OffsetIndexMetaData]>,
+) -> u64 {
     row_group
         .columns()
         .iter()
         .enumerate()
         .filter(|(leaf_index, _)| projection.leaf_included(*leaf_index))
-        .filter_map(|(_, column)| u64::try_from(column.uncompressed_size()).ok())
+        .filter_map(|(leaf_index, column)| {
+            let uncompressed_bytes = u64::try_from(column.uncompressed_size()).ok()?;
+            let selected_bytes = selection
+                .zip(offset_index.and_then(|index| index.get(leaf_index)))
+                .and_then(|(selection, page_index)| {
+                    let page_locations = page_index.page_locations();
+                    let column_start = u64::try_from(
+                        column
+                            .dictionary_page_offset()
+                            .unwrap_or_else(|| column.data_page_offset()),
+                    )
+                    .ok()?;
+                    let dictionary_bytes = u64::try_from(page_locations.first()?.offset)
+                        .ok()?
+                        .checked_sub(column_start)?;
+                    let selected_compressed_bytes = selection
+                        .scan_ranges(page_locations)
+                        .into_iter()
+                        .try_fold(dictionary_bytes, |total, range| {
+                            total.checked_add(range.end.checked_sub(range.start)?)
+                        })?;

Review Comment:
   [P2] Include retained coalesced buffers in sparse read admission
   
   `scan_ranges` sums the selected pages before `ArrowFileReader` merges byte ranges separated by at most 1 MiB. Its returned `Bytes` slices retain the larger coalesced allocations, but the new concurrent row-group admission uses only the smaller selected-page estimate. This leaves live source buffers out of the memory estimate used to increase parallelism.
   
   I verified this with four valid Parquet row groups of 16,384 rows × 128 floats, default Zstd/page settings, and one selected row every 4,096 rows. With a 20,971,520-byte budget, all four groups were admitted and simultaneously retained 25,390,984 bytes of owned read buffers. Allocation ownership and release were tracked with `Bytes::from_owner` and `Drop`; these counts exclude decoded Arrow arrays, and all tracked buffers were released on completion. Please account for the coalesced allocations when calculating admission costs, or retain conservative full-column admission when they cannot be estimated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #800:
URL: https://github.com/apache/paimon-rust/pull/800


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #800:
URL: https://github.com/apache/paimon-rust/pull/800#discussion_r3976835012


##########
crates/paimon/src/table/vindex_index_build_builder/writer.rs:
##########
@@ -87,245 +93,571 @@ impl<'a> VindexIndexBuildBuilder<'a> {
         let expected_bytes = checked_vector_bytes(row_count_usize, dimension_usize)?;
         let training_vector_count =
             checked_training_vector_count(row_count_usize, options.train_sample_ratio)?;
-        let training_buffer_rows =
-            (VECTOR_BUFFER_BYTES / checked_vector_bytes(1, dimension_usize)?).max(1);
-        let training_buffer_floats = training_buffer_rows
-            .checked_mul(dimension_usize)
-            .ok_or_else(|| Error::DataInvalid {
-                message: "vindex training buffer length overflows usize".to_string(),
-                source: None,
-            })?;
+        let training_rows_retained = if self.index_type == DISKANN_IDENTIFIER {
+            0
+        } else {
+            default_training_vector_count(training_vector_count, options.config.nlist()).map_err(
+                |e| Error::DataInvalid {
+                    message: format!("Failed to calculate IVF training vector count: {e}"),
+                    source: Some(Box::new(e)),
+                },
+            )?
+        };
+        let mut sparse_ranges =
+            if training_rows_retained > 0 && training_rows_retained < row_count_usize {
+                Some(plan_ivf_training_ranges(shard, training_rows_retained)?)
+            } else {
+                None
+            };
+        let capability_start = (timing_enabled && sparse_ranges.is_some()).then(Instant::now);
+        if let Some(ranges) = sparse_ranges.as_ref() {
+            let mut checks = Vec::new();
+            let mut usable = true;
+            for file in &shard.files {
+                let path = file.data_file_path(&shard.bucket_path);
+                if !path.to_ascii_lowercase().ends_with(".parquet") {
+                    usable = false;
+                    break;
+                }
+                let Some((file_start, file_end)) = file.row_id_range() else {
+                    usable = false;
+                    break;
+                };
+                let local_ranges = ranges
+                    .iter()
+                    .filter_map(|range| {
+                        let from = range.from().max(file_start);
+                        let to = range.to().min(file_end);
+                        (from <= to).then(|| RowRange::new(from - file_start, to - file_start))
+                    })
+                    .collect::<Vec<_>>();
+                if local_ranges.is_empty()
+                    || file
+                        .write_cols
+                        .as_ref()
+                        .is_some_and(|columns| !columns.iter().any(|column| column == index_column))
+                {
+                    continue;
+                }
+                let file_size = u64::try_from(file.file_size).map_err(|e| Error::DataInvalid {
+                    message: format!(
+                        "Invalid data file size for '{}': {}",
+                        file.file_name, file.file_size
+                    ),
+                    source: Some(Box::new(e)),
+                })?;
+                checks.push((path, file_size, local_ranges));
+            }
+            let found_vector_file = !checks.is_empty();
+            if usable && found_vector_file {
+                let concurrency = self
+                    .table
+                    .schema()
+                    .core_options()
+                    .parquet_row_group_parallelism()?;
+                let file_io = self.table.file_io();
+                let mut checks = futures::stream::iter(checks)
+                    .map(|(path, file_size, local_ranges)| async move {
+                        let input = file_io.new_input(&path)?;
+                        has_usable_offset_index(
+                            Box::new(input.reader().await?),
+                            file_size,
+                            index_column,
+                            &local_ranges,
+                        )
+                        .await

Review Comment:
   [P2] Check actual page savings before enabling the extra source pass
   
   An offset index being present does not mean these ranges avoid reading pages. With the default 100,000-row shard, 65,536 training rows, 128-dimensional vectors, and default Zstd/page settings, the 64 sample ranges leave gaps too small to skip pages or survive the reader's existing 1 MiB range coalescing. A tracking FileRead measured exactly 47,788,791 data bytes for both the sample and the full read, excluding metadata; the offset-index check still returned true. This fixture fits the default writer-buffer and file-size limits.
   
   The subsequent full scan therefore doubles source data reads relative to the previous single source scan, including remote reads when the files are on OSS. This measures bytes, not overall wall time versus the saved local spill. Please estimate selected page ranges after coalescing and fall back, or adapt the sample plan, when the sample would read essentially the whole projection.



##########
crates/paimon/src/table/vindex_index_build_builder/planning.rs:
##########
@@ -45,3 +49,79 @@ pub(super) fn plan_vindex_shards(
         "vindex",
     )
 }
+
+pub(super) fn plan_ivf_training_ranges(
+    shard: &VindexIndexShard,
+    training_rows: usize,
+) -> Result<Vec<RowRange>> {
+    let shard_rows = usize::try_from(checked_row_count(
+        shard.row_range_start,
+        shard.row_range_end,
+    )?)
+    .map_err(|error| Error::DataInvalid {
+        message: "vindex shard row count does not fit usize".to_string(),
+        source: Some(Box::new(error)),
+    })?;
+    if training_rows == 0 || training_rows > shard_rows {
+        return Err(Error::DataInvalid {
+            message: format!(
+                "Invalid IVF training row count: {training_rows}; shard contains {shard_rows} rows"
+            ),
+            source: None,
+        });
+    }
+    if training_rows == shard_rows {
+        return Ok(Vec::new());
+    }
+
+    let range_count = training_rows.min(MAX_IVF_TRAINING_RANGES);
+    let gap_count = range_count + 1;
+    let skipped_rows = shard_rows - training_rows;
+    let seed = mix_seed(
+        (shard.snapshot_id as u64)
+            ^ (shard.row_range_start as u64).rotate_left(21)
+            ^ (shard.row_range_end as u64).rotate_left(42),
+    );
+    let range_extra_offset = seed as usize % range_count;
+    let gap_extra_offset = seed.rotate_left(17) as usize % gap_count;
+    let mut cursor = shard.row_range_start;
+    let mut ranges = Vec::with_capacity(range_count);
+
+    for gap_index in 0..range_count {
+        let gap = skipped_rows / gap_count
+            + usize::from(
+                (gap_index + gap_count - gap_extra_offset) % gap_count < skipped_rows % gap_count,
+            );
+        cursor = checked_add_offset(cursor, gap, "training gap")?;

Review Comment:
   [P2] Preserve representative training samples across the shard
   
   These fixed gaps permanently exclude contiguous regions; the seed only shifts the remainder allocation by approximately one row. For a 100,000-row shard with the default sample ratio of 1 and 65,536 retained vectors, the last sampled row is 99,468, so a newly appended distribution in the final 500 rows is never trained.
   
   I reproduced this using the actual planner and vindex 0.4 trainer: 90,000 vectors `[0]`, then 9,500 `[1]`, then 500 `[100]`, with IVF-SQ, L2, and `nlist=nprobe=1`. Querying `[100]` returns 10/10 results from the final cluster with the previous full-stream reservoir sampling, but 0/10 with these ranges. The SQ bounds are trained only on the older distributions, so the new vectors are clamped to the old upper bound even though every vector is subsequently added and every list is searched. Please retain the existing reservoir path until representative sampling is preserved, and add a recall regression test for data clustered by append order.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #800:
URL: https://github.com/apache/paimon-rust/pull/800#discussion_r4043858471


##########
crates/paimon/src/table/vindex_index_build_builder/pipeline.rs:
##########
@@ -0,0 +1,1113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::extraction::{
+    data_split_for_shard_ranges, extract_vector_batch, validate_vector_batch_ranges,
+};
+use super::planning::VindexIndexShard;
+use super::timing::{vector_index_build_timing_enabled, VectorIndexBuildTiming};
+use super::validation::{
+    checked_row_count, checked_training_sample_index, checked_training_vector_count,
+    checked_vector_bytes,
+};
+use super::writer::BuiltIndexFile;
+use super::VindexIndexBuildBuilder;
+use crate::arrow::format::parquet::{
+    coalesced_parquet_range_bytes, parquet_granules, ParquetGranule,
+};
+use crate::spec::ROW_ID_FIELD_NAME;
+use crate::table::data_file_reader::DataFileReadTiming;
+use crate::table::table_read::configured_parquet_read_budget;
+use crate::table::{merge_row_ranges, ArrowRecordBatchStream, RowRange};
+use crate::vindex::VindexVectorIndexOptions;
+use crate::{Error, Result};
+use arrow_array::RecordBatch;
+use arrow_buffer::MutableBuffer;
+use futures::{StreamExt, TryStreamExt};
+use paimon_vindex_core::autotune::default_training_vector_count;
+use paimon_vindex_core::index::{VectorIndexTrainer, VectorIndexTraining, VectorIndexWriter};
+use std::collections::{HashMap, HashSet};
+use std::io::{BufReader, BufWriter, Read, Seek, SeekFrom, Write};
+use std::ops::Range;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+use tokio::sync::mpsc;
+use tokio::task::JoinHandle;
+
+const MIN_STRATA: usize = 256;
+const ROWS_PER_STRATUM: usize = 128;
+const FIRST_BYTES_NUMERATOR: u64 = 3;
+const FIRST_BYTES_DENOMINATOR: u64 = 10;
+const QUEUE_CAPACITY: usize = 2;
+const BUFFER_BYTES: usize = 8 * 1024 * 1024;
+const REPLAY_TARGET_BYTES: usize = 32 * 1024 * 1024;
+
+#[derive(Clone, Debug)]
+struct Granule {
+    range: RowRange,
+    file_index: usize,
+    byte_ranges: Vec<Range<u64>>,
+}
+
+#[derive(Debug)]
+pub(super) struct GranulePlan {
+    pub(super) first: Vec<RowRange>,
+    pub(super) rest: Vec<RowRange>,
+    first_rows: usize,
+}
+
+struct SpillRecord {
+    ids: Vec<i64>,
+    bytes: Vec<u8>,
+}
+
+enum AddItem {
+    Batch(RecordBatch, Vec<i64>),
+    Spilled(Vec<i64>, MutableBuffer),
+}
+
+type SpillTask = JoinHandle<std::io::Result<(std::fs::File, u64, Duration)>>;
+type ConsumerTask = JoinHandle<Result<(VectorIndexWriter, usize, usize, Duration)>>;
+type TrainingTask = JoinHandle<std::io::Result<(VectorIndexTraining, Duration)>>;
+type ReplayTask = JoinHandle<std::io::Result<(usize, Duration)>>;
+
+struct SpillWriter {
+    sender: mpsc::Sender<SpillRecord>,
+    task: SpillTask,
+}
+
+impl SpillWriter {
+    async fn finish(self) -> Result<(std::fs::File, u64, Duration)> {
+        drop(self.sender);
+        join_spill(self.task).await
+    }
+}
+
+struct LivePipeline {
+    sender: mpsc::Sender<AddItem>,
+    consumer: ConsumerTask,
+    replay: ReplayTask,
+    spill_bytes: u64,
+    spill_write: Duration,
+}
+
+fn spawn_spill_writer(timing_enabled: bool) -> Result<SpillWriter> {
+    let file = tempfile::tempfile().map_err(|e| Error::UnexpectedError {
+        message: format!("Failed to create temporary vindex vector file: {e}"),
+        source: Some(Box::new(e)),
+    })?;
+    let (sender, mut receiver) = mpsc::channel::<SpillRecord>(QUEUE_CAPACITY);
+    let task = tokio::task::spawn_blocking(move || -> std::io::Result<_> {
+        let mut writer = BufWriter::with_capacity(BUFFER_BYTES, file);
+        let mut spill_bytes = 0u64;
+        let mut spill_write = Duration::ZERO;
+        while let Some(record) = receiver.blocking_recv() {
+            let write_start = timing_enabled.then(Instant::now);
+            let count = record.ids.len() as u64;
+            writer.write_all(&count.to_le_bytes())?;
+            for id in record.ids {
+                writer.write_all(&id.to_le_bytes())?;
+            }
+            writer.write_all(&record.bytes)?;
+            spill_bytes = spill_bytes
+                .saturating_add(8)
+                .saturating_add(count.saturating_mul(8))
+                .saturating_add(record.bytes.len() as u64);
+            if let Some(start) = write_start {
+                spill_write = spill_write.saturating_add(start.elapsed());
+            }
+        }
+        let write_start = timing_enabled.then(Instant::now);
+        writer.flush()?;
+        let file = writer.into_inner().map_err(|e| e.into_error())?;
+        if let Some(start) = write_start {
+            spill_write = spill_write.saturating_add(start.elapsed());
+        }
+        Ok((file, spill_bytes, spill_write))
+    });
+    Ok(SpillWriter { sender, task })
+}
+
+async fn join_spill(task: SpillTask) -> Result<(std::fs::File, u64, Duration)> {
+    task.await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex spill task failed: {e}"),
+            source: None,
+        })?
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("Failed to spill vindex vectors: {e}"),
+            source: Some(Box::new(e)),
+        })
+}
+
+fn spawn_add_consumer(
+    writer: VectorIndexWriter,
+    mut receiver: mpsc::Receiver<AddItem>,
+    index_column: String,
+    dimension: usize,
+    timing_enabled: bool,
+) -> ConsumerTask {
+    tokio::task::spawn_blocking(move || -> Result<_> {
+        let mut writer = writer;
+        let mut rows_added = 0usize;
+        let mut replay_rows = 0usize;
+        let mut index_add = Duration::ZERO;
+        while let Some(item) = receiver.blocking_recv() {
+            let add_start = timing_enabled.then(Instant::now);
+            match item {
+                AddItem::Batch(batch, ids) => {
+                    let vectors = extract_vector_batch(&batch, &index_column, dimension)?;
+                    if ids.len() != vectors.row_count {
+                        return Err(Error::DataInvalid {
+                            message: "vindex add batch id count mismatch".to_string(),
+                            source: None,
+                        });
+                    }
+                    writer
+                        .add_vectors(&ids, vectors.values, vectors.row_count)
+                        .map_err(|e| Error::UnexpectedError {
+                            message: format!("Failed to add vectors to vindex index: {e}"),
+                            source: Some(Box::new(e)),
+                        })?;
+                    rows_added += vectors.row_count;
+                }
+                AddItem::Spilled(ids, buffer) => {
+                    let values = buffer.typed_data::<f32>();
+                    if values.len() != ids.len() * dimension {
+                        return Err(Error::DataInvalid {
+                            message: "vindex spilled vector length mismatch".to_string(),
+                            source: None,
+                        });
+                    }
+                    writer.add_vectors(&ids, values, ids.len()).map_err(|e| {
+                        Error::UnexpectedError {
+                            message: format!("Failed to add spilled vectors to vindex index: {e}"),
+                            source: Some(Box::new(e)),
+                        }
+                    })?;
+                    rows_added += ids.len();
+                    replay_rows += ids.len();
+                }
+            }
+            if let Some(start) = add_start {
+                index_add = index_add.saturating_add(start.elapsed());
+            }
+        }
+        Ok((writer, rows_added, replay_rows, index_add))
+    })
+}
+
+async fn join_consumer(task: ConsumerTask) -> Result<(VectorIndexWriter, usize, usize, Duration)> {
+    task.await.map_err(|e| Error::UnexpectedError {
+        message: format!("vindex add task failed: {e}"),
+        source: None,
+    })?
+}
+
+async fn join_training(task: TrainingTask) -> Result<(VectorIndexTraining, Duration)> {
+    task.await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex training task failed: {e}"),
+            source: None,
+        })?
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("Failed to train vindex index: {e}"),
+            source: Some(Box::new(e)),
+        })
+}
+
+async fn start_live_pipeline(
+    training: TrainingTask,
+    spill: SpillWriter,
+    index_column: String,
+    dimension: usize,
+    timing_enabled: bool,
+) -> Result<(LivePipeline, Duration)> {
+    let trained = join_training(training).await;
+    let spilled = spill.finish().await;
+    let (trained, train_finish) = trained?;
+    let (file, spill_bytes, spill_write) = spilled?;
+    let (sender, receiver) = mpsc::channel(QUEUE_CAPACITY);
+    let consumer = spawn_add_consumer(
+        VectorIndexWriter::new(trained),
+        receiver,
+        index_column,
+        dimension,
+        timing_enabled,
+    );
+    let replay = spawn_replay(file, sender.clone(), dimension, timing_enabled);
+    Ok((
+        LivePipeline {
+            sender,
+            consumer,
+            replay,
+            spill_bytes,
+            spill_write,
+        },
+        train_finish,
+    ))
+}
+
+async fn finish_live_pipeline(
+    pipeline: LivePipeline,
+) -> (
+    Result<(VectorIndexWriter, usize, usize, Duration)>,
+    Result<(usize, Duration)>,
+    u64,
+    Duration,
+) {
+    let replay = pipeline
+        .replay
+        .await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex replay task failed: {e}"),
+            source: None,
+        })
+        .and_then(|result| {
+            result.map_err(|e| Error::UnexpectedError {
+                message: format!("Failed to replay spilled vindex vectors: {e}"),
+                source: Some(Box::new(e)),
+            })
+        });
+    drop(pipeline.sender);
+    let consumer = join_consumer(pipeline.consumer).await;
+    (consumer, replay, pipeline.spill_bytes, pipeline.spill_write)
+}
+
+fn spawn_replay(
+    mut file: std::fs::File,
+    sender: mpsc::Sender<AddItem>,
+    dimension: usize,
+    timing_enabled: bool,
+) -> ReplayTask {
+    tokio::task::spawn_blocking(move || -> std::io::Result<(usize, Duration)> {
+        let mut spill_read = Duration::ZERO;
+        let read_start = timing_enabled.then(Instant::now);
+        file.seek(SeekFrom::Start(0))?;
+        if let Some(start) = read_start {
+            spill_read = spill_read.saturating_add(start.elapsed());
+        }
+        let mut reader = BufReader::with_capacity(BUFFER_BYTES, file);
+        let mut rows = 0usize;
+        let mut ids = Vec::new();
+        let mut vectors = MutableBuffer::new(REPLAY_TARGET_BYTES);
+        loop {
+            let read_start = timing_enabled.then(Instant::now);
+            let mut header = [0u8; 8];
+            let read = reader.read(&mut header)?;
+            if read == 0 {
+                if let Some(start) = read_start {
+                    spill_read = spill_read.saturating_add(start.elapsed());
+                }
+                break;
+            }
+            reader.read_exact(&mut header[read..])?;
+            let count = usize::try_from(u64::from_le_bytes(header)).map_err(|_| {
+                std::io::Error::new(std::io::ErrorKind::InvalidData, "invalid spill row count")
+            })?;
+            let id_bytes_len = count.checked_mul(8).ok_or_else(|| {
+                std::io::Error::new(std::io::ErrorKind::InvalidData, "spill id length overflow")
+            })?;
+            let mut id_bytes = vec![0u8; id_bytes_len];
+            reader.read_exact(&mut id_bytes)?;
+            let (id_chunks, remainder) = id_bytes.as_chunks::<8>();
+            debug_assert!(remainder.is_empty());
+            ids.extend(id_chunks.iter().map(|bytes| i64::from_le_bytes(*bytes)));
+            let vector_bytes = count
+                .checked_mul(dimension)
+                .and_then(|value| value.checked_mul(4))
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidData,
+                        "spill vector length overflow",
+                    )
+                })?;
+            let offset = vectors.len();
+            vectors.resize(offset + vector_bytes, 0);
+            reader.read_exact(&mut vectors.as_slice_mut()[offset..])?;
+            if let Some(start) = read_start {
+                spill_read = spill_read.saturating_add(start.elapsed());
+            }
+            rows += count;
+            if vectors.len() >= REPLAY_TARGET_BYTES {
+                let item = AddItem::Spilled(
+                    std::mem::take(&mut ids),
+                    std::mem::replace(&mut vectors, MutableBuffer::new(REPLAY_TARGET_BYTES)),
+                );
+                if sender.blocking_send(item).is_err() {
+                    return Ok((rows, spill_read));
+                }
+            }
+        }
+        if !ids.is_empty() {
+            let _ = sender.blocking_send(AddItem::Spilled(ids, vectors));
+        }
+        Ok((rows, spill_read))
+    })
+}
+
+fn pick_indices(total: usize, count: usize) -> impl Iterator<Item = usize> {
+    (0..count).map(move |index| ((2 * index + 1) * total) / (2 * count))

Review Comment:
   [P2] Avoid fixed midpoint granule sampling that can exclude an entire distribution
   
   Verified on `f934e8a0fdae78d382adaffe2b05696e30070398`, using the PR's locked `paimon-vindex-core 0.4.0` dependency.
   
   The midpoint selection here is deterministic systematic sampling, rather than randomized stratified sampling. For 4,096 granules and a target of 512, it always selects zero-based granule indices `4, 12, 20, ...`. Data correlated with file/page order can therefore be completely absent from training. The trainer's subsequent reservoir sampling cannot recover a distribution that never reaches it.
   
   I reproduced this through the actual granule planner, Parquet reads, index build/serialization, and search, not just a simulation of the selection formula:
   
   - 4,096 real Parquet files in the memory filesystem, 256 rows per file, totaling 1,048,576 rows in one shard.
   - Every eighth file (zero-based file index `% 8 == 7`) contains `[100.0]`; the other files alternate `[0.0]` and `[1.0]`. The excluded cluster represents 12.5% of all rows.
   - IVF-SQ, L2, `dimension=1`, `nlist=nprobe=1`, and `train.sample-ratio=1.0`.
   - The planner selects 512 first ranges / 131,072 candidate rows, containing no `[100.0]` vectors.
   - With `vindex.build.granule.enabled=false`, querying `[100.0]` returns 10/10 results from that cluster. With it enabled, the same query returns 0/10. These are cluster-hit counts, not exact-ID recall, since distances tie within a cluster.
   
   All rows are still added to the index; the regression is in training quality. SQ learns only the `[0, 1]` range and clips the unseen `[100]` vectors to the same upper code. I also reproduced a tail-only distribution failure in a separate actual-planner-plus-trainer comparison.
   
   Could we avoid fixed granule positions and add a regression test for periodic/file-correlated distributions? One conservative option is seeded randomized stratification over logical row positions, expanding selected rows to pages for I/O without treating every fetched row as a training sample. If the required page reads eliminate the savings, retain the full-shard sampling fallback. Randomizing whole granules is less intrusive, but still needs explicit evaluation of rare-cluster coverage and unequal granule sizes. Until that trade-off is validated, I would keep the conservative path as the default.
   
   The existing 24 vindex-build tests and 58 Parquet tests pass; they do not cover this case.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #800:
URL: https://github.com/apache/paimon-rust/pull/800#discussion_r3986223913


##########
crates/paimon/src/table/vindex_index_build_builder/planning.rs:
##########
@@ -45,3 +49,79 @@ pub(super) fn plan_vindex_shards(
         "vindex",
     )
 }
+
+pub(super) fn plan_ivf_training_ranges(
+    shard: &VindexIndexShard,
+    training_rows: usize,
+) -> Result<Vec<RowRange>> {
+    let shard_rows = usize::try_from(checked_row_count(
+        shard.row_range_start,
+        shard.row_range_end,
+    )?)
+    .map_err(|error| Error::DataInvalid {
+        message: "vindex shard row count does not fit usize".to_string(),
+        source: Some(Box::new(error)),
+    })?;
+    if training_rows == 0 || training_rows > shard_rows {
+        return Err(Error::DataInvalid {
+            message: format!(
+                "Invalid IVF training row count: {training_rows}; shard contains {shard_rows} rows"
+            ),
+            source: None,
+        });
+    }
+    if training_rows == shard_rows {
+        return Ok(Vec::new());
+    }
+
+    let range_count = training_rows.min(MAX_IVF_TRAINING_RANGES);
+    let gap_count = range_count + 1;
+    let skipped_rows = shard_rows - training_rows;
+    let seed = mix_seed(
+        (shard.snapshot_id as u64)
+            ^ (shard.row_range_start as u64).rotate_left(21)
+            ^ (shard.row_range_end as u64).rotate_left(42),
+    );
+    let range_extra_offset = seed as usize % range_count;
+    let gap_extra_offset = seed.rotate_left(17) as usize % gap_count;
+    let mut cursor = shard.row_range_start;
+    let mut ranges = Vec::with_capacity(range_count);
+
+    for gap_index in 0..range_count {
+        let gap = skipped_rows / gap_count
+            + usize::from(
+                (gap_index + gap_count - gap_extra_offset) % gap_count < skipped_rows % gap_count,
+            );
+        cursor = checked_add_offset(cursor, gap, "training gap")?;

Review Comment:
   Verified at 28d6670: both original tail-distribution cases (100,000 rows with a final 500-row cluster, and 1,000,000 rows with a final 10,000-row cluster) now return 10/10 correct-cluster results. The specific fixed-gap regression described above is addressed. A separate small-sample case still fails and is documented on the current planner line here: https://github.com/apache/paimon-rust/pull/800#discussion_r3986221481



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(vindex): optimize IVF index build reads [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #800:
URL: https://github.com/apache/paimon-rust/pull/800#discussion_r4055847318


##########
crates/paimon/src/table/vindex_index_build_builder/pipeline.rs:
##########
@@ -0,0 +1,1258 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::extraction::{
+    data_split_for_shard_ranges, extract_vector_batch, validate_vector_batch_ranges,
+};
+use super::planning::VindexIndexShard;
+use super::timing::{vector_index_build_timing_enabled, VectorIndexBuildTiming};
+use super::validation::{
+    checked_row_count, checked_training_sample_index, checked_training_vector_count,
+    checked_vector_bytes,
+};
+use super::writer::BuiltIndexFile;
+use super::VindexIndexBuildBuilder;
+use crate::arrow::format::parquet::{
+    coalesced_parquet_range_bytes, parquet_granules, ParquetGranule,
+};
+use crate::spec::ROW_ID_FIELD_NAME;
+use crate::table::data_file_reader::DataFileReadTiming;
+use crate::table::table_read::configured_parquet_read_budget;
+use crate::table::{merge_row_ranges, ArrowRecordBatchStream, RowRange};
+use crate::vindex::VindexVectorIndexOptions;
+use crate::{Error, Result};
+use arrow_array::RecordBatch;
+use arrow_buffer::MutableBuffer;
+use futures::{StreamExt, TryStreamExt};
+use paimon_vindex_core::autotune::default_training_vector_count;
+use paimon_vindex_core::index::{VectorIndexTrainer, VectorIndexTraining, VectorIndexWriter};
+use std::collections::{HashMap, HashSet};
+use std::io::{BufReader, BufWriter, Read, Seek, SeekFrom, Write};
+use std::ops::Range;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+use tokio::sync::mpsc;
+use tokio::task::JoinHandle;
+
+const MIN_STRATA: usize = 256;
+const ROWS_PER_STRATUM: usize = 128;
+const FIRST_BYTES_NUMERATOR: u64 = 3;
+const FIRST_BYTES_DENOMINATOR: u64 = 10;
+const QUEUE_CAPACITY: usize = 2;
+const BUFFER_BYTES: usize = 8 * 1024 * 1024;
+const REPLAY_TARGET_BYTES: usize = 32 * 1024 * 1024;
+
+#[derive(Clone, Debug)]
+struct Granule {
+    range: RowRange,
+    file_index: usize,
+    byte_ranges: Vec<Range<u64>>,
+}
+
+#[derive(Debug)]
+pub(super) struct GranulePlan {
+    pub(super) first: Vec<RowRange>,
+    pub(super) rest: Vec<RowRange>,
+    first_rows: usize,
+}
+
+struct SpillRecord {
+    ids: Vec<i64>,
+    bytes: Vec<u8>,
+}
+
+enum AddItem {
+    Batch(RecordBatch, Vec<i64>),
+    Spilled(Vec<i64>, MutableBuffer),
+}
+
+type SpillTask = JoinHandle<std::io::Result<(std::fs::File, u64, Duration)>>;
+type ConsumerTask = JoinHandle<Result<(VectorIndexWriter, usize, usize, Duration)>>;
+type TrainingTask = JoinHandle<std::io::Result<(VectorIndexTraining, Duration)>>;
+type ReplayTask = JoinHandle<std::io::Result<(usize, Duration)>>;
+
+struct SpillWriter {
+    sender: mpsc::Sender<SpillRecord>,
+    task: SpillTask,
+}
+
+impl SpillWriter {
+    async fn finish(self) -> Result<(std::fs::File, u64, Duration)> {
+        drop(self.sender);
+        join_spill(self.task).await
+    }
+}
+
+struct LivePipeline {
+    sender: mpsc::Sender<AddItem>,
+    consumer: ConsumerTask,
+    replay: ReplayTask,
+    spill_bytes: u64,
+    spill_write: Duration,
+}
+
+fn spawn_spill_writer(timing_enabled: bool) -> Result<SpillWriter> {
+    let file = tempfile::tempfile().map_err(|e| Error::UnexpectedError {
+        message: format!("Failed to create temporary vindex vector file: {e}"),
+        source: Some(Box::new(e)),
+    })?;
+    let (sender, mut receiver) = mpsc::channel::<SpillRecord>(QUEUE_CAPACITY);
+    let task = tokio::task::spawn_blocking(move || -> std::io::Result<_> {
+        let mut writer = BufWriter::with_capacity(BUFFER_BYTES, file);
+        let mut spill_bytes = 0u64;
+        let mut spill_write = Duration::ZERO;
+        while let Some(record) = receiver.blocking_recv() {
+            let write_start = timing_enabled.then(Instant::now);
+            let count = record.ids.len() as u64;
+            writer.write_all(&count.to_le_bytes())?;
+            for id in record.ids {
+                writer.write_all(&id.to_le_bytes())?;
+            }
+            writer.write_all(&record.bytes)?;
+            spill_bytes = spill_bytes
+                .saturating_add(8)
+                .saturating_add(count.saturating_mul(8))
+                .saturating_add(record.bytes.len() as u64);
+            if let Some(start) = write_start {
+                spill_write = spill_write.saturating_add(start.elapsed());
+            }
+        }
+        let write_start = timing_enabled.then(Instant::now);
+        writer.flush()?;
+        let file = writer.into_inner().map_err(|e| e.into_error())?;
+        if let Some(start) = write_start {
+            spill_write = spill_write.saturating_add(start.elapsed());
+        }
+        Ok((file, spill_bytes, spill_write))
+    });
+    Ok(SpillWriter { sender, task })
+}
+
+async fn join_spill(task: SpillTask) -> Result<(std::fs::File, u64, Duration)> {
+    task.await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex spill task failed: {e}"),
+            source: None,
+        })?
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("Failed to spill vindex vectors: {e}"),
+            source: Some(Box::new(e)),
+        })
+}
+
+fn spawn_add_consumer(
+    writer: VectorIndexWriter,
+    mut receiver: mpsc::Receiver<AddItem>,
+    index_column: String,
+    dimension: usize,
+    timing_enabled: bool,
+) -> ConsumerTask {
+    tokio::task::spawn_blocking(move || -> Result<_> {
+        let mut writer = writer;
+        let mut rows_added = 0usize;
+        let mut replay_rows = 0usize;
+        let mut index_add = Duration::ZERO;
+        while let Some(item) = receiver.blocking_recv() {
+            let add_start = timing_enabled.then(Instant::now);
+            match item {
+                AddItem::Batch(batch, ids) => {
+                    let vectors = extract_vector_batch(&batch, &index_column, dimension)?;
+                    if ids.len() != vectors.row_count {
+                        return Err(Error::DataInvalid {
+                            message: "vindex add batch id count mismatch".to_string(),
+                            source: None,
+                        });
+                    }
+                    writer
+                        .add_vectors(&ids, vectors.values, vectors.row_count)
+                        .map_err(|e| Error::UnexpectedError {
+                            message: format!("Failed to add vectors to vindex index: {e}"),
+                            source: Some(Box::new(e)),
+                        })?;
+                    rows_added += vectors.row_count;
+                }
+                AddItem::Spilled(ids, buffer) => {
+                    let values = buffer.typed_data::<f32>();
+                    if values.len() != ids.len() * dimension {
+                        return Err(Error::DataInvalid {
+                            message: "vindex spilled vector length mismatch".to_string(),
+                            source: None,
+                        });
+                    }
+                    writer.add_vectors(&ids, values, ids.len()).map_err(|e| {
+                        Error::UnexpectedError {
+                            message: format!("Failed to add spilled vectors to vindex index: {e}"),
+                            source: Some(Box::new(e)),
+                        }
+                    })?;
+                    rows_added += ids.len();
+                    replay_rows += ids.len();
+                }
+            }
+            if let Some(start) = add_start {
+                index_add = index_add.saturating_add(start.elapsed());
+            }
+        }
+        Ok((writer, rows_added, replay_rows, index_add))
+    })
+}
+
+async fn join_consumer(task: ConsumerTask) -> Result<(VectorIndexWriter, usize, usize, Duration)> {
+    task.await.map_err(|e| Error::UnexpectedError {
+        message: format!("vindex add task failed: {e}"),
+        source: None,
+    })?
+}
+
+async fn join_training(task: TrainingTask) -> Result<(VectorIndexTraining, Duration)> {
+    task.await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex training task failed: {e}"),
+            source: None,
+        })?
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("Failed to train vindex index: {e}"),
+            source: Some(Box::new(e)),
+        })
+}
+
+async fn start_live_pipeline(
+    training: TrainingTask,
+    spill: SpillWriter,
+    index_column: String,
+    dimension: usize,
+    timing_enabled: bool,
+) -> Result<(LivePipeline, Duration)> {
+    let trained = join_training(training).await;
+    let spilled = spill.finish().await;
+    let (trained, train_finish) = trained?;
+    let (file, spill_bytes, spill_write) = spilled?;
+    let (sender, receiver) = mpsc::channel(QUEUE_CAPACITY);
+    let consumer = spawn_add_consumer(
+        VectorIndexWriter::new(trained),
+        receiver,
+        index_column,
+        dimension,
+        timing_enabled,
+    );
+    let replay = spawn_replay(file, sender.clone(), dimension, timing_enabled);
+    Ok((
+        LivePipeline {
+            sender,
+            consumer,
+            replay,
+            spill_bytes,
+            spill_write,
+        },
+        train_finish,
+    ))
+}
+
+async fn finish_live_pipeline(
+    pipeline: LivePipeline,
+) -> (
+    Result<(VectorIndexWriter, usize, usize, Duration)>,
+    Result<(usize, Duration)>,
+    u64,
+    Duration,
+) {
+    let replay = pipeline
+        .replay
+        .await
+        .map_err(|e| Error::UnexpectedError {
+            message: format!("vindex replay task failed: {e}"),
+            source: None,
+        })
+        .and_then(|result| {
+            result.map_err(|e| Error::UnexpectedError {
+                message: format!("Failed to replay spilled vindex vectors: {e}"),
+                source: Some(Box::new(e)),
+            })
+        });
+    drop(pipeline.sender);
+    let consumer = join_consumer(pipeline.consumer).await;
+    (consumer, replay, pipeline.spill_bytes, pipeline.spill_write)
+}
+
+fn spawn_replay(
+    mut file: std::fs::File,
+    sender: mpsc::Sender<AddItem>,
+    dimension: usize,
+    timing_enabled: bool,
+) -> ReplayTask {
+    tokio::task::spawn_blocking(move || -> std::io::Result<(usize, Duration)> {
+        let mut spill_read = Duration::ZERO;
+        let read_start = timing_enabled.then(Instant::now);
+        file.seek(SeekFrom::Start(0))?;
+        if let Some(start) = read_start {
+            spill_read = spill_read.saturating_add(start.elapsed());
+        }
+        let mut reader = BufReader::with_capacity(BUFFER_BYTES, file);
+        let mut rows = 0usize;
+        let mut ids = Vec::new();
+        let mut vectors = MutableBuffer::new(REPLAY_TARGET_BYTES);
+        loop {
+            let read_start = timing_enabled.then(Instant::now);
+            let mut header = [0u8; 8];
+            let read = reader.read(&mut header)?;
+            if read == 0 {
+                if let Some(start) = read_start {
+                    spill_read = spill_read.saturating_add(start.elapsed());
+                }
+                break;
+            }
+            reader.read_exact(&mut header[read..])?;
+            let count = usize::try_from(u64::from_le_bytes(header)).map_err(|_| {
+                std::io::Error::new(std::io::ErrorKind::InvalidData, "invalid spill row count")
+            })?;
+            let id_bytes_len = count.checked_mul(8).ok_or_else(|| {
+                std::io::Error::new(std::io::ErrorKind::InvalidData, "spill id length overflow")
+            })?;
+            let mut id_bytes = vec![0u8; id_bytes_len];
+            reader.read_exact(&mut id_bytes)?;
+            let (id_chunks, remainder) = id_bytes.as_chunks::<8>();
+            debug_assert!(remainder.is_empty());
+            ids.extend(id_chunks.iter().map(|bytes| i64::from_le_bytes(*bytes)));
+            let vector_bytes = count
+                .checked_mul(dimension)
+                .and_then(|value| value.checked_mul(4))
+                .ok_or_else(|| {
+                    std::io::Error::new(
+                        std::io::ErrorKind::InvalidData,
+                        "spill vector length overflow",
+                    )
+                })?;
+            let offset = vectors.len();
+            vectors.resize(offset + vector_bytes, 0);
+            reader.read_exact(&mut vectors.as_slice_mut()[offset..])?;
+            if let Some(start) = read_start {
+                spill_read = spill_read.saturating_add(start.elapsed());
+            }
+            rows += count;
+            if vectors.len() >= REPLAY_TARGET_BYTES {
+                let item = AddItem::Spilled(
+                    std::mem::take(&mut ids),
+                    std::mem::replace(&mut vectors, MutableBuffer::new(REPLAY_TARGET_BYTES)),
+                );
+                if sender.blocking_send(item).is_err() {
+                    return Ok((rows, spill_read));
+                }
+            }
+        }
+        if !ids.is_empty() {
+            let _ = sender.blocking_send(AddItem::Spilled(ids, vectors));
+        }
+        Ok((rows, spill_read))
+    })
+}
+
+fn splitmix64(state: &mut u64) -> u64 {
+    *state = state.wrapping_add(0x9E37_79B9_7F4A_7C15);
+    let mut z = *state;
+    z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+    z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+    z ^ (z >> 31)
+}
+
+fn shard_seed(shard: &VindexIndexShard) -> u64 {
+    let mut seed = 0u64;
+    let mut absorb = |value: u64| {
+        let mut state = seed ^ value;
+        seed = splitmix64(&mut state);
+    };
+    absorb(shard.snapshot_id as u64);
+    absorb(shard.source_bucket as u64);
+    absorb(shard.row_range_start as u64);
+    absorb(shard.row_range_end as u64);
+    absorb(shard.partition_bytes.len() as u64);
+    for chunk in shard.partition_bytes.chunks(8) {
+        let mut word = [0u8; 8];
+        word[..chunk.len()].copy_from_slice(chunk);
+        absorb(u64::from_le_bytes(word));
+    }
+    seed
+}
+
+fn pick_indices(total: usize, count: usize, seed: u64) -> impl Iterator<Item = usize> {
+    let mut state = seed;
+    (0..count).map(move |index| {
+        let start = index * total / count;
+        let end = (index + 1) * total / count;
+        start + (splitmix64(&mut state) % (end - start) as u64) as usize
+    })
+}
+
+fn granule_bytes(granules: &[Granule], selected: Option<&HashSet<usize>>) -> u64 {
+    let mut files = HashMap::<usize, Vec<Range<u64>>>::new();
+    for (index, granule) in granules.iter().enumerate() {
+        if selected.is_none_or(|selected| selected.contains(&index)) {
+            files
+                .entry(granule.file_index)
+                .or_default()
+                .extend(granule.byte_ranges.iter().cloned());
+        }
+    }
+    files
+        .values()
+        .map(|ranges| coalesced_parquet_range_bytes(ranges))
+        .sum()
+}
+
+fn select_first(granules: &[Granule], training_rows: usize, seed: u64) -> HashSet<usize> {
+    let mut target = training_rows
+        .div_ceil(ROWS_PER_STRATUM)
+        .max(MIN_STRATA)
+        .min(granules.len());
+    let (selected, rows) = loop {
+        let selected = pick_indices(granules.len(), target, seed).collect::<HashSet<_>>();
+        let rows = selected
+            .iter()
+            .map(|index| granules[*index].range.count() as usize)
+            .sum::<usize>();
+        if rows >= training_rows || target == granules.len() {
+            break (selected, rows);
+        }
+        target = target
+            .saturating_mul(training_rows)
+            .div_ceil(rows.max(1))
+            .clamp(target + 1, granules.len());
+    };
+    let first_bytes = granule_bytes(granules, Some(&selected));
+    let total_bytes = granule_bytes(granules, None);
+    if selected.len() < MIN_STRATA.min(granules.len())
+        || rows < training_rows
+        || total_bytes == 0
+        || first_bytes.saturating_mul(FIRST_BYTES_DENOMINATOR)
+            > total_bytes.saturating_mul(FIRST_BYTES_NUMERATOR)
+    {
+        (0..granules.len()).collect()
+    } else {
+        selected
+    }
+}
+
+fn append_shard_granules(
+    granules: &mut Vec<Granule>,
+    file_index: usize,
+    file_start: i64,
+    shard_range: &RowRange,
+    file_granules: Vec<ParquetGranule>,
+) -> Result<()> {
+    for granule in file_granules {
+        let from = file_start
+            .checked_add(granule.first_row)
+            .ok_or_else(|| Error::DataInvalid {
+                message: "vindex granule row id overflows i64".to_string(),
+                source: None,
+            })?;
+        let to = from
+            .checked_add(granule.row_count - 1)
+            .ok_or_else(|| Error::DataInvalid {
+                message: "vindex granule row range overflows i64".to_string(),
+                source: None,
+            })?;
+        if let Some(range) = shard_range.intersect_inclusive(from, to) {
+            granules.push(Granule {
+                range,
+                file_index,
+                byte_ranges: granule.byte_ranges,
+            });
+        }
+    }
+    Ok(())
+}
+
+fn granules_partition_shard(granules: &[Granule], shard_range: &RowRange) -> bool {
+    if granules
+        .windows(2)
+        .any(|pair| pair[1].range.from() <= pair[0].range.to())
+    {
+        return false;
+    }
+    let coverage = merge_row_ranges(
+        granules
+            .iter()
+            .map(|granule| granule.range.clone())
+            .collect(),
+    );
+    coverage.len() == 1 && coverage.first() == Some(shard_range)
+}
+
+fn local_ids(row_ids: &[i64], start: i64, row_count: usize) -> Result<Vec<i64>> {
+    let end = start
+        .checked_add(i64::try_from(row_count).map_err(|e| Error::DataInvalid {
+            message: "vindex row count does not fit i64".to_string(),
+            source: Some(Box::new(e)),
+        })?)
+        .ok_or_else(|| Error::DataInvalid {
+            message: "vindex row range overflows i64".to_string(),
+            source: None,
+        })?;
+    row_ids
+        .iter()
+        .map(|row_id| {
+            if *row_id < start || *row_id >= end {
+                Err(Error::DataInvalid {
+                    message: format!("vindex row id {row_id} is outside shard [{start}, {end})"),
+                    source: None,
+                })
+            } else {
+                Ok(*row_id - start)
+            }
+        })
+        .collect()
+}
+
+impl<'a> VindexIndexBuildBuilder<'a> {
+    fn open_vector_stream(
+        &self,
+        shard: &VindexIndexShard,
+        ranges: Vec<RowRange>,
+        index_column: &str,
+        read_timing: Option<&Arc<DataFileReadTiming>>,
+        parquet_read_budget: Option<&Arc<crate::arrow::ReadBudget>>,
+    ) -> Result<ArrowRecordBatchStream> {
+        let split = data_split_for_shard_ranges(shard, ranges)?;
+        let mut read_builder = self.table.new_read_builder();
+        read_builder.with_projection(&[index_column, ROW_ID_FIELD_NAME])?;
+        let read = read_builder.new_read()?;
+        let read = match read_timing {
+            Some(timing) => read.with_data_file_read_timing(Arc::clone(timing)),
+            None => read,
+        };
+        let read = match parquet_read_budget {
+            Some(budget) => read.with_parquet_read_budget(Arc::clone(budget)),
+            None => read,
+        };
+        read.to_arrow(&[split])
+    }
+
+    pub(super) async fn plan_granules(
+        &self,
+        shard: &VindexIndexShard,
+        index_column: &str,
+        training_rows: usize,
+    ) -> Result<GranulePlan> {
+        let shard_range = RowRange::new(shard.row_range_start, shard.row_range_end);
+        let mut granules = Vec::new();
+        let mut use_whole_shard = false;
+        let mut parquet_files = Vec::new();
+
+        for (file_index, file) in shard.files.iter().enumerate() {
+            if file
+                .write_cols
+                .as_ref()
+                .is_some_and(|columns| !columns.iter().any(|column| column == index_column))
+            {
+                continue;
+            }
+            let Some((file_start, file_end)) = file.row_id_range() else {
+                use_whole_shard = true;
+                break;
+            };
+            let Some(range) = shard_range.intersect_inclusive(file_start, file_end) else {
+                continue;
+            };
+            let file_size = u64::try_from(file.file_size).map_err(|e| Error::DataInvalid {
+                message: format!(
+                    "Invalid data file size for '{}': {}",
+                    file.file_name, file.file_size
+                ),
+                source: Some(Box::new(e)),
+            })?;
+            let path = file.data_file_path(&shard.bucket_path);
+            if path.to_ascii_lowercase().ends_with(".parquet") {
+                parquet_files.push((file_index, path, file_size, file_start, file_end));
+            } else {
+                granules.push(Granule {
+                    range,
+                    file_index,
+                    byte_ranges: std::iter::once(0..file_size).collect(),
+                });
+            }
+        }
+
+        if !use_whole_shard && !parquet_files.is_empty() {
+            let concurrency = self
+                .table
+                .schema()
+                .core_options()
+                .parquet_row_group_parallelism()?
+                .max(1);
+            let file_io = self.table.file_io();
+            let mut results = futures::stream::iter(parquet_files)
+                .map(
+                    |(file_index, path, file_size, file_start, file_end)| async move {
+                        let input = file_io.new_input(&path)?;
+                        let reader = Box::new(input.reader().await?);
+                        let (granules, _) =
+                            parquet_granules(reader, file_size, index_column).await?;
+                        Ok::<_, Error>((file_index, file_start, file_end, granules))
+                    },
+                )
+                .buffer_unordered(concurrency);
+            while let Some(result) = results.next().await {
+                match result {
+                    Ok((file_index, file_start, file_end, file_granules)) => {
+                        let covered = file_granules
+                            .iter()
+                            .map(|granule| granule.row_count)
+                            .sum::<i64>();
+                        if covered != file_end - file_start + 1 {
+                            use_whole_shard = true;
+                            break;
+                        }
+                        append_shard_granules(
+                            &mut granules,
+                            file_index,
+                            file_start,
+                            &shard_range,
+                            file_granules,
+                        )?;
+                    }
+                    Err(error) => {
+                        log::warn!(
+                            "vindex granule metadata read failed; using the whole shard as the first granule: {error}"
+                        );
+                        use_whole_shard = true;
+                        break;
+                    }
+                }
+            }
+        }
+
+        granules.sort_by_key(|granule| granule.range.from());
+        if !use_whole_shard {
+            // Data Evolution may retain multiple physical providers for the same logical rows.
+            // Let the reader choose the provider instead of counting those rows twice here.
+            use_whole_shard = !granules_partition_shard(&granules, &shard_range);
+        }
+        if use_whole_shard || granules.is_empty() {
+            granules = vec![Granule {
+                range: shard_range,
+                file_index: 0,
+                byte_ranges: std::iter::once(
+                    0..shard
+                        .files
+                        .iter()
+                        .map(|file| file.file_size.max(0) as u64)
+                        .sum(),
+                )
+                .collect(),
+            }];
+        }
+
+        let selected = select_first(&granules, training_rows, shard_seed(shard));
+        let mut first = Vec::with_capacity(selected.len());
+        let mut rest = Vec::with_capacity(granules.len() - selected.len());
+        let mut first_rows = 0usize;
+        for (index, granule) in granules.iter().enumerate() {
+            if selected.contains(&index) {
+                first_rows += granule.range.count() as usize;
+                first.push(granule.range.clone());
+            } else {
+                rest.push(granule.range.clone());
+            }
+        }
+        Ok(GranulePlan {
+            first: merge_row_ranges(first),
+            rest: merge_row_ranges(rest),
+            first_rows,
+        })
+    }
+
+    #[allow(clippy::too_many_arguments)]
+    pub(super) async fn build_index_file_granule(
+        &self,
+        shard: &VindexIndexShard,
+        index_column: &str,
+        dimension: i32,
+        index_field_id: i32,
+        options: &VindexVectorIndexOptions,
+        index_meta: Vec<u8>,
+    ) -> Result<BuiltIndexFile> {
+        let timing_enabled = vector_index_build_timing_enabled();
+        let total_start = timing_enabled.then(Instant::now);
+        let mut source_batch_wait = Duration::ZERO;
+        let mut batch_count = 0usize;
+        let read_timing = timing_enabled.then(|| Arc::new(DataFileReadTiming::default()));
+        let parquet_read_budget = if timing_enabled {
+            let budget = configured_parquet_read_budget(self.table)?;
+            budget.enable_diagnostics();
+            Some(budget)
+        } else {
+            None
+        };
+        let row_count = checked_row_count(shard.row_range_start, shard.row_range_end)?;
+        let row_count_usize = usize::try_from(row_count).map_err(|e| Error::DataInvalid {
+            message: format!("Invalid vindex row count: {row_count}"),
+            source: Some(Box::new(e)),
+        })?;
+        let dimension = usize::try_from(dimension).map_err(|e| Error::DataInvalid {
+            message: format!("Invalid vindex dimension: {dimension}"),
+            source: Some(Box::new(e)),
+        })?;
+        if dimension == 0 {
+            return Err(Error::DataInvalid {
+                message: "vindex vector dimension must be positive".to_string(),
+                source: None,
+            });
+        }
+        checked_vector_bytes(row_count_usize, dimension)?;
+        let eligible = checked_training_vector_count(row_count_usize, options.train_sample_ratio)?;
+        let retained =
+            default_training_vector_count(eligible, options.config.nlist()).unwrap_or(eligible);
+        let plan = self.plan_granules(shard, index_column, retained).await?;
+
+        let mut trainer =
+            VectorIndexTrainer::new(options.config.clone()).map_err(|e| Error::DataInvalid {
+                message: format!("Failed to initialize vindex trainer: {e}"),
+                source: Some(Box::new(e)),
+            })?;
+        let mut spill = Some(spawn_spill_writer(timing_enabled)?);
+        let training_rows = eligible.min(plan.first_rows);
+        let training_buffer_rows = (BUFFER_BYTES / checked_vector_bytes(1, dimension)?).max(1);
+        let training_buffer_floats = training_buffer_rows * dimension;
+        let mut training_buffer = Vec::with_capacity(training_buffer_floats);
+        let mut next_training_sample = 0usize;
+        let mut first_rows = 0usize;
+        let mut range_index = 0usize;
+        let first_result: Result<()> = async {
+            let mut stream = self.open_vector_stream(
+                shard,
+                plan.first.clone(),
+                index_column,
+                read_timing.as_ref(),
+                parquet_read_budget.as_ref(),
+            )?;
+            let mut expected_row_id = plan.first[0].from();
+            loop {
+                let source_start = timing_enabled.then(Instant::now);
+                let batch = stream.try_next().await?;
+                if let Some(start) = source_start {
+                    source_batch_wait = source_batch_wait.saturating_add(start.elapsed());
+                }
+                let Some(batch) = batch else { break };
+                batch_count += 1;
+                let vectors = validate_vector_batch_ranges(
+                    &batch,
+                    index_column,
+                    dimension,
+                    &plan.first,
+                    &mut range_index,
+                    &mut expected_row_id,
+                )?;
+                let batch_end = first_rows.checked_add(vectors.row_count).ok_or_else(|| {
+                    Error::DataInvalid {
+                        message: "vindex first-batch row count overflows usize".to_string(),
+                        source: None,
+                    }
+                })?;
+                while next_training_sample < training_rows {
+                    let sample = checked_training_sample_index(
+                        next_training_sample,
+                        plan.first_rows,
+                        training_rows,
+                    )?;
+                    if sample >= batch_end {
+                        break;
+                    }
+                    let offset = (sample - first_rows) * dimension;
+                    training_buffer.extend_from_slice(&vectors.values[offset..offset + dimension]);
+                    next_training_sample += 1;
+                    if training_buffer.len() == training_buffer_floats {
+                        trainer
+                            .add_training_vectors_mut(
+                                &training_buffer,
+                                training_buffer.len() / dimension,
+                            )
+                            .map_err(|e| Error::DataInvalid {
+                                message: format!("Failed to add vindex training vectors: {e}"),
+                                source: Some(Box::new(e)),
+                            })?;
+                        training_buffer.clear();
+                    }
+                }
+                let record = SpillRecord {
+                    ids: local_ids(vectors.row_ids, shard.row_range_start, row_count_usize)?,
+                    bytes: vectors.bytes.to_vec(),
+                };
+                first_rows = batch_end;
+                if spill.as_ref().unwrap().sender.send(record).await.is_err() {
+                    return Err(Error::UnexpectedError {
+                        message: "vindex spill writer stopped unexpectedly".to_string(),
+                        source: None,
+                    });
+                }
+            }
+            if !training_buffer.is_empty() {
+                trainer
+                    .add_training_vectors_mut(&training_buffer, training_buffer.len() / dimension)
+                    .map_err(|e| Error::DataInvalid {
+                        message: format!("Failed to add vindex training vectors: {e}"),
+                        source: Some(Box::new(e)),
+                    })?;
+            }
+            if first_rows != plan.first_rows
+                || range_index != plan.first.len()
+                || next_training_sample != training_rows
+            {
+                return Err(Error::DataInvalid {
+                    message: format!(
+                        "vindex first-batch mismatch: rows={first_rows}/{}, ranges={range_index}/{}, training={next_training_sample}/{training_rows}",
+                        plan.first_rows,
+                        plan.first.len()
+                    ),
+                    source: None,
+                });
+            }
+            Ok(())
+        }
+        .await;
+        if let Err(error) = first_result {
+            return match spill.take().unwrap().finish().await {
+                Ok(_) => Err(error),
+                Err(spill_error) => Err(spill_error),
+            };
+        }
+
+        let mut training: Option<TrainingTask> = Some(tokio::task::spawn_blocking(
+            move || -> std::io::Result<_> {
+                let start = timing_enabled.then(Instant::now);
+                let training = trainer.finish()?;
+                Ok((
+                    training,
+                    start.map_or(Duration::ZERO, |start| start.elapsed()),
+                ))
+            },
+        ));
+        let mut live: Option<LivePipeline> = None;
+        let mut train_finish = Duration::ZERO;
+        let index_column = index_column.to_string();
+
+        macro_rules! go_live {
+            () => {{
+                let (pipeline, duration) = start_live_pipeline(
+                    training.take().expect("training task"),
+                    spill.take().expect("spill writer"),
+                    index_column.clone(),
+                    dimension,
+                    timing_enabled,
+                )
+                .await?;
+                train_finish = duration;
+                live = Some(pipeline);
+            }};
+        }
+
+        let mut rest_rows = 0usize;
+        let producer_result: Result<()> = async {
+            if !plan.rest.is_empty() {
+                let mut stream = self.open_vector_stream(
+                    shard,
+                    plan.rest.clone(),
+                    &index_column,
+                    read_timing.as_ref(),
+                    parquet_read_budget.as_ref(),
+                )?;
+                let mut range_index = 0usize;
+                let mut expected_row_id = plan.rest[0].from();
+                loop {
+                    let source_start = timing_enabled.then(Instant::now);
+                    let batch = stream.try_next().await?;
+                    if let Some(start) = source_start {
+                        source_batch_wait = source_batch_wait.saturating_add(start.elapsed());
+                    }
+                    let Some(batch) = batch else { break };
+                    batch_count += 1;
+                    if live.is_none() && training.as_ref().is_some_and(|task| task.is_finished()) {
+                        go_live!();
+                    }
+                    let vectors = validate_vector_batch_ranges(
+                        &batch,
+                        &index_column,
+                        dimension,
+                        &plan.rest,
+                        &mut range_index,
+                        &mut expected_row_id,
+                    )?;
+                    let ids = local_ids(vectors.row_ids, shard.row_range_start, row_count_usize)?;
+                    rest_rows = rest_rows.checked_add(vectors.row_count).ok_or_else(|| {
+                        Error::DataInvalid {
+                            message: "vindex remaining row count overflows usize".to_string(),
+                            source: None,
+                        }
+                    })?;
+                    if let Some(pipeline) = &live {
+                        if pipeline
+                            .sender
+                            .send(AddItem::Batch(batch, ids))
+                            .await
+                            .is_err()
+                        {
+                            return Err(Error::UnexpectedError {
+                                message: "vindex add consumer stopped unexpectedly".to_string(),
+                                source: None,
+                            });
+                        }
+                    } else {
+                        let record = SpillRecord {
+                            ids,
+                            bytes: vectors.bytes.to_vec(),
+                        };
+                        if spill.as_ref().unwrap().sender.send(record).await.is_err() {
+                            return Err(Error::UnexpectedError {
+                                message: "vindex spill writer stopped unexpectedly".to_string(),
+                                source: None,
+                            });
+                        }
+                    }
+                }
+                if rest_rows != row_count_usize - plan.first_rows
+                    || range_index != plan.rest.len()
+                {
+                    return Err(Error::DataInvalid {
+                        message: format!(
+                            "vindex remaining-batch mismatch: rows={rest_rows}/{}, ranges={range_index}/{}",
+                            row_count_usize - plan.first_rows,
+                            plan.rest.len()
+                        ),
+                        source: None,
+                    });
+                }
+            }
+            Ok(())
+        }
+        .await;
+
+        if let Err(producer_error) = producer_result {
+            if let Some(pipeline) = live.take() {
+                let (consumer, replay, _, _) = finish_live_pipeline(pipeline).await;
+                consumer?;
+                replay?;
+            } else {
+                let spill_result = match spill.take() {
+                    Some(spill) => spill.finish().await.map(|_| ()),
+                    None => Ok(()),
+                };
+                let training_result = match training.take() {
+                    Some(training) => join_training(training).await.map(|_| ()),
+                    None => Ok(()),
+                };
+                spill_result?;
+                training_result?;
+            }
+            return Err(producer_error);
+        }
+        if live.is_none() {
+            go_live!();
+        }
+
+        let (consumer, replay, granule_spill_bytes, granule_spill_write) =
+            finish_live_pipeline(live.unwrap()).await;
+        let (writer, rows_added, consumer_replay_rows, index_add) = consumer?;
+        let (replay_rows, granule_spill_read) = replay?;
+        if rows_added != row_count_usize || replay_rows != consumer_replay_rows {
+            return Err(Error::DataInvalid {
+                message: format!(
+                    "vindex pipelined add mismatch: rows={rows_added}/{row_count_usize}, replay={consumer_replay_rows}/{replay_rows}"
+                ),
+                source: None,
+            });
+        }
+
+        let serialize_upload_start = timing_enabled.then(Instant::now);
+        let meta = self
+            .finish_index_file(writer, shard, index_field_id, index_meta, row_count)
+            .await?;
+        let serialize_upload =
+            serialize_upload_start.map_or(Duration::ZERO, |start| start.elapsed());
+        let (oss_read, parquet_decode) = read_timing
+            .as_ref()
+            .map_or((Duration::ZERO, Duration::ZERO), |timing| {
+                (timing.file_read(), timing.parquet_decode())
+            });
+        let (file_schema_open, first_batch_wait, remaining_batch_wait) = read_timing
+            .as_ref()
+            .map_or((Duration::ZERO, Duration::ZERO, Duration::ZERO), |timing| {
+                timing.file_waits()
+            });
+        let parquet_diagnostics = parquet_read_budget
+            .as_ref()
+            .map_or_else(Default::default, |budget| budget.diagnostics());
+        let timing = total_start.map(|start| VectorIndexBuildTiming {
+            total_without_commit: start.elapsed(),
+            source_batch_wait,
+            oss_read,
+            parquet_decode,
+            file_schema_open,
+            first_batch_wait,
+            remaining_batch_wait,
+            parquet_row_group_count: parquet_diagnostics.row_group_count,
+            parquet_projected_bytes_min: parquet_diagnostics.projected_bytes_min,
+            parquet_projected_bytes_max: parquet_diagnostics.projected_bytes_max,
+            parquet_projected_bytes_total: parquet_diagnostics.projected_bytes_total,
+            parquet_peak_inflight_row_groups: parquet_diagnostics.peak_inflight,
+            raw_temp_write: Duration::ZERO,
+            granule_spill_write,
+            train_finish,
+            raw_temp_reread: Duration::ZERO,
+            granule_spill_read,
+            index_add,
+            serialize_upload,
+            rows: row_count_usize,
+            training_rows_seen: plan.first_rows,

Review Comment:
   [P2] Report the actual trainer sample counts
   
   These fields no longer match the established full-spill semantics. Only `training_rows` are passed to `VectorIndexTrainer`, and its reservoir retains the already-computed `retained` count. The current values therefore overstate the metrics on realistic builds: with a coarse 1,000-row granule and `train.sample-ratio=0.1`, the trainer sees 100 rows but `training_rows_seen` reports 1,000; with 10M rows and `nlist=4096`, the core retains 262,144 rows while `training_rows_retained` can report 698,768. This makes the default granule-path diagnostics misleading and prevents reliable comparisons with the full-spill path. Please use `training_rows_seen: training_rows` and `training_rows_retained: retained`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org