[PR] [QDP] Use a VecDeque ring buffer in StreamingProducer (#1436) [mahout]

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman opened a new pull request, #1559:
URL: https://github.com/apache/mahout/pull/1559

   ### Purpose of PR
   #1436 already has an open PR, #1462 by @0lai0 (assigned GSoC work). This PR is offered for comparison only; the author should coordinate on #1462 first, and parts of it (such as the legacy equivalence test) may fit better as review suggestions there. The change replaces the `Vec` + cursor buffer, whose `drain(..cursor)` compaction memmoves the retained tail, with a `VecDeque` whose front drain only advances the head. Output is unchanged.
   
   ### Related Issues or PRs
   Related to #1436 (comparison with #1462, which is linked to close it)
   
   ### Changes Made
   - `qdp-core/src/pipeline_runner.rs`, `StreamingProducer`:
     - buffer is a `VecDeque<T>`; new `fill_buffer` helper appends chunks with `extend(&[T])`;
     - `produce` copies the batch out of `as_slices()` (both halves when it straddles the wrap point) and drops it with `drain(..take)`;
     - `BUFFER_COMPACT_DENOM` and `buffer_cursor` removed;
     - new `streaming_ring` helper reserves `batch + chunk` once (reusing the first chunk's allocation), so the ring does not reallocate afterwards.
   - `vecdeque_streaming_tests` (with a verbatim copy of the old producer as `legacy`):
     - `streaming_vecdeque_matches_legacy_bit_for_bit`: f32/f64 Parquet, read-chunk sizes 4..1024, batch sizes 1..200, with and without recycling;
     - `streaming_ring_capacity_constant_over_100_batches`: capacity constant, output matches legacy, refills and at least one wrapped copy occur.
   
   Compared with #1462 (read-only, not built): same core design. Differences: this PR adds the legacy sweep and a named `streaming_ring` helper; #1462 includes an `#[ignore]`d benchmark, which this PR does not.
   
   ### Testing
   - `cargo test -p qdp-core --lib`: 99 passed, 3 failed (main 97/3; same 3 GPU-only tests; +2 new).
   - `--test parquet_f32 --test reader`: 7 + 3 passed, files unmodified.
   - clippy `-D warnings` (qdp-core + qdp-python), `cargo check -p qdp-python`, `cargo fmt --check`: clean.
   - Not verified: speed (no benchmark run), large real-world Parquet files and 20+ qubit samples, the downstream GPU encode.
   
   ### Merging
   Applies to main on its own and touches only `StreamingProducer`. Conflicts with the #1434/#1435 PRs in `pipeline_runner.rs`; an "after #1435" version (changing `StreamingAdapter`) exists.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_01GuXmXRXdNF43h2FoQcNAZb
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org