[PR] Stream reader search for non-concurrent pread [paimon-vector-index]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #12:
URL: https://github.com/apache/paimon-vector-index/pull/12

   ## Summary
   
   Reduce memory pressure for single-query reader search when the underlying stream cannot do true concurrent positional reads. Such readers now read and scan one probed inverted list at a time instead of pre-reading all probed lists before scanning.
   
   ## Changes
   
   - Add `IVFPQIndexReader::supports_concurrent_pread()` to expose the underlying stream capability.
   - Keep the existing pre-read + parallel scan path for readers that support concurrent `pread`.
   - Use a sequential read-and-scan path for non-concurrent readers so only one probed list needs to be resident at a time.
   - Share list scanning logic through a helper so both paths produce equivalent results.
   - Add a reader-search test for a non-concurrent `pread` stream.
   
   ## Testing
   
   - `cargo fmt --all -- --check`
   - `cargo test -p paimon-vindex-core`
   - `cargo clippy --all-targets --workspace -- -D warnings`
   - `cargo build --all-targets`
   - `cargo test --workspace`
   - `git diff --check`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Stream reader search for non-concurrent pread [paimon-vector-index]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #12:
URL: https://github.com/apache/paimon-vector-index/pull/12#issuecomment-4648139110

   Addressed the requested input-validation fix in 333475c. Reader search now requires query.len() == d, and test_reader_search_validates_inputs covers both short (d - 1) and long (d + 1) queries returning InvalidInput instead of reaching the distance debug assertion.\n\nValidation run locally:\n- cargo fmt --all -- --check\n- cargo test -p paimon-vindex-core reader_search -- --nocapture\n- cargo test -p paimon-vindex-core ivfpq::tests::test_reader_search_validates_inputs -- --nocapture\n- cargo test -p paimon-vindex-core\n- cargo test --workspace\n- cargo clippy --all-targets --workspace -- -D warnings\n- cargo build --all-targets\n- git diff --check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Stream reader search for non-concurrent pread [paimon-vector-index]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #12:
URL: https://github.com/apache/paimon-vector-index/pull/12


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Stream reader search for non-concurrent pread [paimon-vector-index]

["leaves12138 (via GitHub)" <gi...@apache.org>]

leaves12138 commented on code in PR #12:
URL: https://github.com/apache/paimon-vector-index/pull/12#discussion_r3372326771


##########
core/src/ivfpq.rs:
##########
@@ -855,6 +871,29 @@ pub fn search_with_reader_filter<R: SeekRead>(
 ) -> io::Result<(Vec<i64>, Vec<f32>)> {
     reader.ensure_loaded()?;
     let d = reader.d;
+    if query.len() < d {

Review Comment:
   This check should reject `query.len() != d`, not only `query.len() < d`. With a longer query, `q = query.to_vec()` keeps the extra elements, and `kmeans::find_topk` calls `fvec_l2sqr(&q, centroid_slice)`, which hits the `debug_assert_eq!(a.len(), b.len())` in debug builds. I reproduced this by passing a `d + 1` query to `reader.search`; it panics before returning an `InvalidInput` error. The JNI/Python frontends already enforce exact length, but the core `IVFPQIndexReader::search` API should do the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org