You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "pmcgleenon (via GitHub)" <gi...@apache.org> on 2026/08/18 13:43:46 UTC
[PR] REQ sketch : port of reqsketch-rs (datasketches-rust)
pmcgleenon opened a new pull request, #204:
URL: https://github.com/apache/datasketches-rust/pull/204
This is a port of [reqsketch-rs](https://github.com/pmcgleenon/reqsketch-rs) as discussed here https://github.com/apache/datasketches-rust/issues/90. This implementation is influenced heavily by the apache datasketches C++ implementation.
Some things present in reqsketch-rs that are not part of this PR:
- [benchmarks](https://github.com/pmcgleenon/reqsketch-rs/blob/main/benches/core_benchmarks.rs), I left these out since I didn't see any other benchmarks. Maybe we can add some in a later PR for all the sketches
- program build the plots [rank error plots](https://github.com/pmcgleenon/reqsketch-rs/blob/main/examples/req_rank_error.rs). There are a bunch of dependencies needed for this so I left it out. But I think it's a great visualization of the reqsketch algorithm and helps with validation of the results since the apache-datasketches project [has similar plots](https://datasketches.apache.org/docs/REQ/ReqSketch.html) and they can be easily compared to validate the results
@tisonkun FYI if you get a chance please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#issuecomment-5345104139
Glanced at the surface and commented above. Below is a review comment from Codex. I'll dive into implementation details later this week. And I think it's OK to merge a starting implementation and iterate later. But let's discuss existing items first to decide when to converge.
[CODEX COMMENT STARTS]
I found two correctness issues that I believe need to be fixed before this can be merged.
1. **`Compactor::merge` breaks the ordering invariant required by the wire format**
In `datasketches/src/req/compactor.rs`, `Compactor::merge` appends the other compactor’s items and marks the result as unsorted. This is safe only while the sketch remains in memory, because `rank()` falls back to a linear scan.
However, the REQ wire format stores a sorted flag only for level 0. Higher levels are implicitly required to be sorted, and `ReqSketch::deserialize` consequently marks every level above 0 as sorted. A merged sketch can therefore serialize unsorted higher-level items, which are then binary-searched as if they were sorted after deserialization.
I reproduced this with two HRA sketches using `k = 12`, containing `1000..1072` and `0..72`. After merging the latter into the former and performing a serialize/deserialize round trip, the restored sketch returned different results from its sorted view for the same rank query. In one run:
```text
restored.rank(0.0, Inclusive) = 0.097222...
restored.sorted_view().rank(0.0, Inclusive) = 0.013888...
```
This also means a merged image produced by Rust may be interpreted incorrectly by Java or C++.
Both reference implementations preserve ordering during compactor merge by sorting and performing an ordered merge. I suggest doing the same here, including validating that the two compactors have the same `lg_weight`.
Please also add a regression test that:
- builds two estimation-mode sketches with reversed/disjoint value ranges;
- merges them;
- serializes and deserializes the result; and
- verifies that direct rank queries remain equal to sorted-view rank queries.
2. **Deserialization accepts impossible compactor states that later panic or violate public API contracts**
The current validation checks `section_size_raw` and requires `lg_weight < 64`, but it does not validate several structural invariants controlled by the serialized input.
Two concrete examples:
- An image with `num_sections = 0` is accepted. Calling `merge()` on the resulting sketch then panics in `ensure_enough_sections()` when evaluating `num_sections - 1`.
- A single-level image with `lg_weight = 63` and `n = 1` is accepted. Its inclusive rank query returns approximately `9.22e18`, despite `rank()` promising a result in `[0, 1]`.
A malformed serialized image should not be able to create a sketch for which safe public operations panic or return impossible results.
Before constructing the sketch, I suggest validating at least:
- each compactor’s `lg_weight` equals its level index;
- `num_sections`, `section_size_raw`, and `state` form a valid, non-zero compactor configuration;
- the number of levels is representable by the `u64` weight model;
- `sum(num_items * 2^level) == n`;
- levels declared as sorted are actually sorted; and
- extrema and retained values are consistent and contain no invalid NaN state.
The relevant arithmetic should also use checked operations. Since these failures originate from malformed serialized input, they should return `ErrorKind::InvalidData`, not `InvalidArgument`.
Once these two issues are addressed, the existing algorithm and cross-language compatibility coverage will be on much firmer ground.
[CODEX COMMENT END]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814845057
##########
datasketches/src/req/union.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ union — combines REQ sketches into a single result.
+
+use super::RankAccuracy;
+use super::sketch::ReqSketch;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// Combines multiple [`ReqSketch`]es into a single result.
+///
+/// Internally wraps a `ReqSketch` configured for union semantics. The C++
+/// equivalent is `req_union<T>`.
+#[derive(Debug, Clone)]
+pub struct ReqUnion<T: ReqValue> {
+ inner: ReqSketch<T>,
+}
+
+impl<T: ReqValue> ReqUnion<T> {
+ /// Creates a new union with default `k = 12` and `RankAccuracy::HighRank`.
+ pub fn new() -> Self {
+ Self {
+ inner: ReqSketch::new(),
+ }
+ }
+
+ /// Creates a new union with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is invalid (see [`ReqSketch::try_new`]).
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ Ok(Self {
+ inner: ReqSketch::try_new(k, rank_accuracy)?,
+ })
+ }
Review Comment:
Similarly, I'd prefer `new` to be a panic version of `try_new` and leave the no-param `new` as `default`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814852391
##########
datasketches/src/req/union.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ union — combines REQ sketches into a single result.
+
+use super::RankAccuracy;
+use super::sketch::ReqSketch;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// Combines multiple [`ReqSketch`]es into a single result.
+///
+/// Internally wraps a `ReqSketch` configured for union semantics. The C++
+/// equivalent is `req_union<T>`.
Review Comment:
cc @AlexanderSaydakov @leerho You can review the `ReqUnion` design here and give some high-level design comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "jeffoodchain (via GitHub)" <gi...@apache.org>.
jeffoodchain commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3821376563
##########
datasketches/Cargo.toml:
##########
@@ -87,9 +93,14 @@ name = "tuple_test"
path = "tests/tuple_test/main.rs"
required-features = ["tuple"]
+[dependencies]
+rand = { workspace = true, optional = true }
Review Comment:
thanks, I think it's better to have `rand` as dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847071822
##########
datasketches/src/req/compactor.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Compactor implementation for REQ sketch levels.
+//!
+//! Each level in the REQ sketch uses a compactor to maintain a bounded set of items
+//! with deterministic compaction when capacity is exceeded.
+
+use super::RankAccuracy;
+use super::value::ReqValue;
+use crate::error::Error;
+
+fn nearest_even(value: f32) -> u32 {
+ ((value / 2.0).round() as u32) << 1
+}
+
+/// A compactor maintains items at a specific level of the REQ sketch.
+///
+/// When the compactor reaches its nominal capacity, it performs compaction
+/// by keeping approximately half the items and promoting the rest to the next level.
+#[derive(Debug, Clone)]
+pub(super) struct Compactor<T> {
+ /// Current items in the compactor
+ items: Vec<T>,
+ /// Whether items are currently sorted
+ is_sorted: bool,
+ /// State for deterministic compaction
+ state: u64,
+ /// Reusable scratch buffer for compaction operations
+ scratch_buffer: Vec<T>,
+
+ /// Actual section size (rounded to integer)
+ section_size: u32,
+ /// Number of sections in this compactor
+ num_sections: u8,
+ /// The level of this compactor (0 = base level)
+ lg_weight: u8,
+
+ /// Whether this compactor is configured for high rank accuracy
+ rank_accuracy: RankAccuracy,
+ /// Raw section size (may be fractional)
+ section_size_raw: f32,
+ /// Random bit for compaction
+ coin: bool,
+}
+
+impl<T> Compactor<T>
+where
+ T: Clone + ReqValue,
+{
+ /// Creates a new compactor for the given level.
+ ///
+ /// # Arguments
+ /// * `lg_weight` - The level (log weight) of this compactor
+ /// * `k` - The k parameter from the parent sketch
+ /// * `rank_accuracy` - Rank accuracy configuration
+ pub(super) fn new(lg_weight: u8, k: u16, rank_accuracy: RankAccuracy) -> Self {
+ let section_size_raw = k as f32;
+ let section_size = nearest_even(section_size_raw);
+ let num_sections = 3u8;
+
+ let nominal: usize = (2 * section_size * num_sections as u32) as usize;
+
+ Self {
+ items: Vec::with_capacity(nominal),
+ is_sorted: true,
+ state: 0,
+ scratch_buffer: Vec::with_capacity(nominal / 2 + 8),
+
+ section_size,
+ num_sections,
+ lg_weight,
+
+ rank_accuracy,
+ section_size_raw,
+ coin: false,
+ }
+ }
+
+ /// Returns the number of items currently in this compactor.
+ pub(super) fn num_items(&self) -> u32 {
+ self.items.len() as u32
+ }
+
+ /// Returns the nominal capacity of this compactor.
+ pub(super) fn nominal_capacity(&self) -> u32 {
+ 2 * self.section_size * self.num_sections as u32
+ }
+
+ /// Returns whether the items are currently sorted.
+ pub(super) fn is_sorted(&self) -> bool {
+ self.is_sorted
+ }
+
+ /// Appends an item to this compactor.
+ #[inline(always)]
+ pub(super) fn append(&mut self, item: T) {
+ self.items.push(item);
+ if self.items.len() > 1 {
+ self.is_sorted = false;
+ }
+ }
+
+ /// Merges items from another compactor into this one.
+ pub(super) fn merge(&mut self, other: &Self) {
+ self.state |= other.state;
+ self.items.extend_from_slice(&other.items);
+ if !other.items.is_empty() {
+ self.is_sorted = false;
+ }
+ // OR-ing the schedule counters can advance state past several doubling
+ // thresholds at once. Loop until no more doublings are needed (C++:
+ // req_compactor_impl.hpp:250 — `while (ensure_enough_sections()) {}`).
+ while self.ensure_enough_sections() {}
+ }
+
+ /// Counts the items at-or-below (`inclusive`) or strictly below `item`.
+ ///
+ /// Uses binary search when this compactor is sorted, and a linear scan
+ /// otherwise. This lets [`ReqSketch::rank`](super::ReqSketch::rank) sum
+ /// per-level weights directly without first building a sorted view.
+ pub(super) fn count_below(&self, item: &T, inclusive: bool) -> usize {
+ if self.is_sorted {
+ if inclusive {
+ self.items.partition_point(|x| x.total_cmp(item).is_le())
+ } else {
+ self.items.partition_point(|x| x.total_cmp(item).is_lt())
+ }
+ } else {
+ self.items
+ .iter()
+ .filter(|x| {
+ let ord = x.total_cmp(item);
+ if inclusive { ord.is_le() } else { ord.is_lt() }
+ })
+ .count()
+ }
+ }
+
+ /// Merges pre-sorted items into this compactor.
+ /// Merges sorted items into this compactor using scratch buffer to avoid allocation.
+ /// Both this compactor's items and the input must be sorted.
+ #[inline(always)]
+ pub(super) fn merge_sorted(&mut self, items: &[T]) {
+ if items.is_empty() {
+ return;
+ }
+
+ if self.items.is_empty() {
+ self.items.extend_from_slice(items);
+ self.is_sorted = true;
+ return;
+ }
+
+ // Ensure sorted on both inputs by contract
+ let total = self.items.len() + items.len();
+ self.scratch_buffer.clear();
+ if self.scratch_buffer.capacity() < total {
+ self.scratch_buffer
+ .reserve(total - self.scratch_buffer.capacity());
+ }
+
+ let (mut i, mut j) = (0usize, 0usize);
+ let (a, b) = (&self.items, items);
+
+ // Two-pointer merge into scratch buffer
+ while i < a.len() && j < b.len() {
+ if a[i].total_cmp(&b[j]).is_le() {
+ self.scratch_buffer.push(a[i].clone());
+ i += 1;
+ } else {
+ self.scratch_buffer.push(b[j].clone());
+ j += 1;
+ }
+ }
+
+ // Add remaining elements
+ if i < a.len() {
+ self.scratch_buffer.extend_from_slice(&a[i..]);
+ }
+ if j < b.len() {
+ self.scratch_buffer.extend_from_slice(&b[j..]);
+ }
+
+ // Swap scratch buffer with items (zero-copy)
+ self.items.clear();
+ std::mem::swap(&mut self.items, &mut self.scratch_buffer);
+ self.is_sorted = true;
+ }
+
+ /// Sorts the items in this compactor if not already sorted.
+ #[inline(always)]
+ pub(super) fn sort(&mut self) {
+ if !self.is_sorted {
+ // Use unstable sort for better performance (stable not needed for REQ sketch)
+ self.items.sort_unstable_by(|a, b| a.total_cmp(b));
+ self.is_sorted = true;
+ }
+ }
+
+ /// Compacts into the provided output buffer without allocating.
+ /// Writes promoted items into `out` and removes the compacted range in-place via `copy_within +
+ /// truncate`.
+ #[inline(always)]
+ pub(super) fn compact_into(&mut self, _rank_accuracy: RankAccuracy, out: &mut Vec<T>) {
+ if self.items.is_empty() {
+ out.clear();
+ return;
+ }
+
+ // Sort entire buffer (C++ sorts full buffer before compaction)
+ self.sort();
+
+ // Calculate sections to compact based on state
+ let secs_to_compact =
+ ((!self.state).trailing_zeros() + 1).min(self.num_sections as u32) as u8;
+ let compaction_range = self.compute_compaction_range(secs_to_compact);
+
+ // Must have at least 2 items to compact
+ if compaction_range.1 <= compaction_range.0 || (compaction_range.1 - compaction_range.0) < 2
+ {
+ out.clear();
+ return;
+ }
+
+ if (self.state & 1) == 1 {
+ self.coin = !self.coin; // flip coin for odd states
+ } else {
+ self.coin = rand::random::<bool>(); // random coin flip for even states
+ }
+ let odds = self.coin;
+
+ // Build promoted items directly into output buffer (no alloc)
+ out.clear();
+ let (start, end) = compaction_range;
+ let mut i = start + if odds { 1 } else { 0 };
+ while i < end {
+ out.push(self.items[i].clone()); // TODO: use Copy fast-path for numeric types
+ i += 2;
+ }
+
+ // Remove the compacted range in-place by rotating elements left
+ let removed = end - start;
+ if end < self.items.len() {
+ // Use rotate_left to move tail elements to fill the gap
+ self.items[start..].rotate_left(removed);
+ }
+ self.items.truncate(self.items.len() - removed);
+
+ // Update state, then ensure enough sections (C++ order)
+ self.state += 1;
+ self.ensure_enough_sections();
+ }
+
+ /// Returns an iterator over the items in this compactor.
+ pub(super) fn iter(&self) -> impl Iterator<Item = &T> {
+ self.items.iter()
+ }
+
+ /// Returns a slice of items for zero-allocation iteration.
+ pub(super) fn items_slice(&self) -> &[T] {
+ &self.items
+ }
+
+ /// Returns the weight (2^lg_weight) for items in this compactor.
+ pub(super) fn weight(&self) -> u64 {
+ 1u64 << self.lg_weight
+ }
+
+ // Private helper methods
+
+ fn ensure_enough_sections(&mut self) -> bool {
+ let ssr = self.section_size_raw / (2.0_f32).sqrt();
Review Comment:
thanks, I've used it here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
##########
datasketches/src/req/compactor.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Compactor implementation for REQ sketch levels.
+//!
+//! Each level in the REQ sketch uses a compactor to maintain a bounded set of items
+//! with deterministic compaction when capacity is exceeded.
+
+use super::RankAccuracy;
+use super::value::ReqValue;
+use crate::error::Error;
+
+fn nearest_even(value: f32) -> u32 {
+ ((value / 2.0).round() as u32) << 1
+}
+
+/// A compactor maintains items at a specific level of the REQ sketch.
+///
+/// When the compactor reaches its nominal capacity, it performs compaction
+/// by keeping approximately half the items and promoting the rest to the next level.
+#[derive(Debug, Clone)]
+pub(super) struct Compactor<T> {
+ /// Current items in the compactor
+ items: Vec<T>,
+ /// Whether items are currently sorted
+ is_sorted: bool,
+ /// State for deterministic compaction
+ state: u64,
+ /// Reusable scratch buffer for compaction operations
+ scratch_buffer: Vec<T>,
+
+ /// Actual section size (rounded to integer)
+ section_size: u32,
+ /// Number of sections in this compactor
+ num_sections: u8,
+ /// The level of this compactor (0 = base level)
+ lg_weight: u8,
+
+ /// Whether this compactor is configured for high rank accuracy
+ rank_accuracy: RankAccuracy,
+ /// Raw section size (may be fractional)
+ section_size_raw: f32,
+ /// Random bit for compaction
+ coin: bool,
+}
+
+impl<T> Compactor<T>
+where
+ T: Clone + ReqValue,
+{
+ /// Creates a new compactor for the given level.
+ ///
+ /// # Arguments
+ /// * `lg_weight` - The level (log weight) of this compactor
+ /// * `k` - The k parameter from the parent sketch
+ /// * `rank_accuracy` - Rank accuracy configuration
+ pub(super) fn new(lg_weight: u8, k: u16, rank_accuracy: RankAccuracy) -> Self {
+ let section_size_raw = k as f32;
+ let section_size = nearest_even(section_size_raw);
+ let num_sections = 3u8;
+
+ let nominal: usize = (2 * section_size * num_sections as u32) as usize;
+
+ Self {
+ items: Vec::with_capacity(nominal),
+ is_sorted: true,
+ state: 0,
+ scratch_buffer: Vec::with_capacity(nominal / 2 + 8),
+
+ section_size,
+ num_sections,
+ lg_weight,
+
+ rank_accuracy,
+ section_size_raw,
+ coin: false,
+ }
+ }
+
+ /// Returns the number of items currently in this compactor.
+ pub(super) fn num_items(&self) -> u32 {
+ self.items.len() as u32
+ }
+
+ /// Returns the nominal capacity of this compactor.
+ pub(super) fn nominal_capacity(&self) -> u32 {
+ 2 * self.section_size * self.num_sections as u32
+ }
+
+ /// Returns whether the items are currently sorted.
+ pub(super) fn is_sorted(&self) -> bool {
+ self.is_sorted
+ }
+
+ /// Appends an item to this compactor.
+ #[inline(always)]
+ pub(super) fn append(&mut self, item: T) {
+ self.items.push(item);
+ if self.items.len() > 1 {
+ self.is_sorted = false;
+ }
+ }
+
+ /// Merges items from another compactor into this one.
+ pub(super) fn merge(&mut self, other: &Self) {
+ self.state |= other.state;
+ self.items.extend_from_slice(&other.items);
+ if !other.items.is_empty() {
+ self.is_sorted = false;
+ }
+ // OR-ing the schedule counters can advance state past several doubling
+ // thresholds at once. Loop until no more doublings are needed (C++:
+ // req_compactor_impl.hpp:250 — `while (ensure_enough_sections()) {}`).
+ while self.ensure_enough_sections() {}
+ }
+
+ /// Counts the items at-or-below (`inclusive`) or strictly below `item`.
+ ///
+ /// Uses binary search when this compactor is sorted, and a linear scan
+ /// otherwise. This lets [`ReqSketch::rank`](super::ReqSketch::rank) sum
+ /// per-level weights directly without first building a sorted view.
+ pub(super) fn count_below(&self, item: &T, inclusive: bool) -> usize {
+ if self.is_sorted {
+ if inclusive {
+ self.items.partition_point(|x| x.total_cmp(item).is_le())
+ } else {
+ self.items.partition_point(|x| x.total_cmp(item).is_lt())
+ }
+ } else {
+ self.items
+ .iter()
+ .filter(|x| {
+ let ord = x.total_cmp(item);
+ if inclusive { ord.is_le() } else { ord.is_lt() }
+ })
+ .count()
+ }
+ }
+
+ /// Merges pre-sorted items into this compactor.
+ /// Merges sorted items into this compactor using scratch buffer to avoid allocation.
+ /// Both this compactor's items and the input must be sorted.
+ #[inline(always)]
+ pub(super) fn merge_sorted(&mut self, items: &[T]) {
+ if items.is_empty() {
+ return;
+ }
+
+ if self.items.is_empty() {
+ self.items.extend_from_slice(items);
+ self.is_sorted = true;
+ return;
+ }
+
+ // Ensure sorted on both inputs by contract
+ let total = self.items.len() + items.len();
+ self.scratch_buffer.clear();
+ if self.scratch_buffer.capacity() < total {
+ self.scratch_buffer
+ .reserve(total - self.scratch_buffer.capacity());
+ }
+
+ let (mut i, mut j) = (0usize, 0usize);
+ let (a, b) = (&self.items, items);
+
+ // Two-pointer merge into scratch buffer
+ while i < a.len() && j < b.len() {
+ if a[i].total_cmp(&b[j]).is_le() {
+ self.scratch_buffer.push(a[i].clone());
+ i += 1;
+ } else {
+ self.scratch_buffer.push(b[j].clone());
+ j += 1;
+ }
+ }
+
+ // Add remaining elements
+ if i < a.len() {
+ self.scratch_buffer.extend_from_slice(&a[i..]);
+ }
+ if j < b.len() {
+ self.scratch_buffer.extend_from_slice(&b[j..]);
+ }
+
+ // Swap scratch buffer with items (zero-copy)
+ self.items.clear();
+ std::mem::swap(&mut self.items, &mut self.scratch_buffer);
+ self.is_sorted = true;
+ }
+
+ /// Sorts the items in this compactor if not already sorted.
+ #[inline(always)]
+ pub(super) fn sort(&mut self) {
+ if !self.is_sorted {
+ // Use unstable sort for better performance (stable not needed for REQ sketch)
+ self.items.sort_unstable_by(|a, b| a.total_cmp(b));
+ self.is_sorted = true;
+ }
+ }
+
+ /// Compacts into the provided output buffer without allocating.
+ /// Writes promoted items into `out` and removes the compacted range in-place via `copy_within +
+ /// truncate`.
+ #[inline(always)]
+ pub(super) fn compact_into(&mut self, _rank_accuracy: RankAccuracy, out: &mut Vec<T>) {
+ if self.items.is_empty() {
+ out.clear();
+ return;
+ }
+
+ // Sort entire buffer (C++ sorts full buffer before compaction)
+ self.sort();
+
+ // Calculate sections to compact based on state
+ let secs_to_compact =
+ ((!self.state).trailing_zeros() + 1).min(self.num_sections as u32) as u8;
+ let compaction_range = self.compute_compaction_range(secs_to_compact);
+
+ // Must have at least 2 items to compact
+ if compaction_range.1 <= compaction_range.0 || (compaction_range.1 - compaction_range.0) < 2
+ {
+ out.clear();
+ return;
+ }
+
+ if (self.state & 1) == 1 {
+ self.coin = !self.coin; // flip coin for odd states
+ } else {
+ self.coin = rand::random::<bool>(); // random coin flip for even states
+ }
+ let odds = self.coin;
+
+ // Build promoted items directly into output buffer (no alloc)
+ out.clear();
+ let (start, end) = compaction_range;
+ let mut i = start + if odds { 1 } else { 0 };
+ while i < end {
+ out.push(self.items[i].clone()); // TODO: use Copy fast-path for numeric types
+ i += 2;
+ }
+
+ // Remove the compacted range in-place by rotating elements left
+ let removed = end - start;
+ if end < self.items.len() {
+ // Use rotate_left to move tail elements to fill the gap
+ self.items[start..].rotate_left(removed);
+ }
+ self.items.truncate(self.items.len() - removed);
+
+ // Update state, then ensure enough sections (C++ order)
+ self.state += 1;
+ self.ensure_enough_sections();
+ }
+
+ /// Returns an iterator over the items in this compactor.
+ pub(super) fn iter(&self) -> impl Iterator<Item = &T> {
+ self.items.iter()
+ }
+
+ /// Returns a slice of items for zero-allocation iteration.
+ pub(super) fn items_slice(&self) -> &[T] {
+ &self.items
+ }
+
+ /// Returns the weight (2^lg_weight) for items in this compactor.
+ pub(super) fn weight(&self) -> u64 {
+ 1u64 << self.lg_weight
+ }
+
+ // Private helper methods
+
+ fn ensure_enough_sections(&mut self) -> bool {
+ let ssr = self.section_size_raw / (2.0_f32).sqrt();
+ let ne = nearest_even(ssr);
+
+ const MIN_K: u32 = 4; // matches datasketches-cpp
Review Comment:
Good catch, updated https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814737183
##########
datasketches/src/codec/family.rs:
##########
@@ -80,6 +80,15 @@ impl Family {
max_pre_longs: 5,
};
+ /// Relative Error Quantiles (REQ) sketch.
+ #[cfg(feature = "req")]
+ pub const REQ: Family = Family {
+ id: 17,
+ name: "REQ",
+ min_pre_longs: 2,
+ max_pre_longs: 4,
Review Comment:
From datasketches-java, these fields are:
```java
REQ(17, "REQ", 1 /* minPreLongs */, 2 /* maxPreLongs */),
```
But we don't use these fields for REQSketch anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814753940
##########
datasketches/Cargo.toml:
##########
@@ -87,9 +93,14 @@ name = "tuple_test"
path = "tests/tuple_test/main.rs"
required-features = ["tuple"]
+[dependencies]
+rand = { workspace = true, optional = true }
Review Comment:
This can be relevant to https://github.com/apache/datasketches-rust/issues/203.
cc @jeffoodchain
We may use `rand` as a starting point and conclude before the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814820764
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
Other builders use a `panic` flavor on setters.
I may prefer to follow the same flavor and propose a new flavor if desired later.
Besides, once we check `k` here, we may not need to call `ReqSketch::try_new` on build which check the same conditions.
And we can have a `ReqSketch::new` as the panic version for `try_new`. The current no-param `new` can be inlined to `default` and only `default`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814742833
##########
datasketches/tests/req_test/serialization.rs:
##########
Review Comment:
This should put under `serde_test` and need not to repeat `serialization_test_data` helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848715259
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
OK. Then let me merge this PR as is and I'll push my follow-ups in a separate PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847067218
##########
datasketches/src/req/union.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ union — combines REQ sketches into a single result.
+
+use super::RankAccuracy;
+use super::sketch::ReqSketch;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// Combines multiple [`ReqSketch`]es into a single result.
+///
+/// Internally wraps a `ReqSketch` configured for union semantics. The C++
+/// equivalent is `req_union<T>`.
Review Comment:
I removed the reference to req_union here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
We could collapse it back into ReqSketch or leave it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848240917
##########
datasketches/Cargo.toml:
##########
@@ -87,9 +93,14 @@ name = "tuple_test"
path = "tests/tuple_test/main.rs"
required-features = ["tuple"]
+[dependencies]
+rand = { workspace = true, optional = true }
Review Comment:
If we do need external deps here, let's start with `rand`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814835517
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
+
+ /// Sets the rank accuracy.
+ pub fn rank_accuracy(mut self, ra: RankAccuracy) -> Self {
+ self.rank_accuracy = ra;
+ self
+ }
+
+ /// Builds the sketch.
+ pub fn build(self) -> Result<ReqSketch<T>, Error> {
+ ReqSketch::try_new(self.k, self.rank_accuracy)
+ }
+}
+
+impl<T: ReqValue + fmt::Display> fmt::Display for ReqSketch<T> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ writeln!(f, "REQ Sketch Summary:")?;
+ writeln!(f, " k : {}", self.k)?;
+ writeln!(f, " rank accuracy : {:?}", self.rank_accuracy)?;
+ writeln!(f, " n : {}", self.n)?;
+ writeln!(f, " num retained : {}", self.num_retained)?;
+ writeln!(f, " num levels : {}", self.compactors.len())?;
+ writeln!(f, " estimation mode : {}", self.is_estimation_mode())?;
+ if let (Some(min), Some(max)) = (&self.min_item, &self.max_item) {
+ writeln!(f, " min item : {min}")?;
+ writeln!(f, " max item : {max}")?;
+ }
+ Ok(())
+ }
+}
Review Comment:
Good point. I wonder if we'd have a tracking issue to `impl fmt::Display` for all sketches when applicible. This is what datasketches-java does IMO, but the format can be considered once more.
cc @notfilippo @ariesdevil @ZENOTME
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814839358
##########
datasketches/src/req/union.rs:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ union — combines REQ sketches into a single result.
+
+use super::RankAccuracy;
+use super::sketch::ReqSketch;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// Combines multiple [`ReqSketch`]es into a single result.
+///
+/// Internally wraps a `ReqSketch` configured for union semantics. The C++
+/// equivalent is `req_union<T>`.
Review Comment:
I see no `req_union<T>` in datasketches-cpp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814774333
##########
datasketches/Cargo.toml:
##########
@@ -87,9 +93,14 @@ name = "tuple_test"
path = "tests/tuple_test/main.rs"
required-features = ["tuple"]
+[dependencies]
+rand = { workspace = true, optional = true }
+
[dev-dependencies]
+approx = { workspace = true }
Review Comment:
`googletest` provides:
* https://docs.rs/googletest/latest/googletest/matchers/fn.near.html
* https://docs.rs/googletest/latest/googletest/matchers/fn.approx_eq.html
.. and we may need one more test lib. But you can leave it to me to do a global alignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847022159
##########
datasketches/src/codec/family.rs:
##########
@@ -80,6 +80,15 @@ impl Family {
max_pre_longs: 5,
};
+ /// Relative Error Quantiles (REQ) sketch.
+ #[cfg(feature = "req")]
+ pub const REQ: Family = Family {
+ id: 17,
+ name: "REQ",
+ min_pre_longs: 2,
+ max_pre_longs: 4,
Review Comment:
Have updated it here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847059651
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
I prefer to avoid run-time crashes for these kind of errors and was taking a similar approach to the C++ error handling which is using exceptions rather than asserts. I'm happy to align with the rest of the sketches here if needed. I think nopanic for this crate would be an improved developer experience, maybe we can consider that for the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#issuecomment-5402878626
> * [benchmarks](https://github.com/pmcgleenon/reqsketch-rs/blob/main/benches/core_benchmarks.rs), I left these out since I didn't see any other benchmarks. Maybe we can add some in a later PR for all the sketches
Now we have one under `datasketches/benches`. If the bench-only dependencies get complicated, we can have a standalone local crate like [apache/asyncband's benchmarks](https://github.com/apache/asyncband/tree/879dc4eec566bc2f210d1a8b924fa78ce0f53189/benchmarks).
But we can introduce the benchmarks in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848680479
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
Reasonable. I'm trying to find a way to merge this PR without too many extended requirements and do a thorough refactor of all related API later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848268788
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
+
+ /// Sets the rank accuracy.
+ pub fn rank_accuracy(mut self, ra: RankAccuracy) -> Self {
+ self.rank_accuracy = ra;
+ self
+ }
+
+ /// Builds the sketch.
+ pub fn build(self) -> Result<ReqSketch<T>, Error> {
+ ReqSketch::try_new(self.k, self.rank_accuracy)
+ }
+}
+
+impl<T: ReqValue + fmt::Display> fmt::Display for ReqSketch<T> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ writeln!(f, "REQ Sketch Summary:")?;
+ writeln!(f, " k : {}", self.k)?;
+ writeln!(f, " rank accuracy : {:?}", self.rank_accuracy)?;
+ writeln!(f, " n : {}", self.n)?;
+ writeln!(f, " num retained : {}", self.num_retained)?;
+ writeln!(f, " num levels : {}", self.compactors.len())?;
+ writeln!(f, " estimation mode : {}", self.is_estimation_mode())?;
+ if let (Some(min), Some(max)) = (&self.min_item, &self.max_item) {
+ writeln!(f, " min item : {min}")?;
+ writeln!(f, " max item : {max}")?;
+ }
+ Ok(())
+ }
+}
Review Comment:
Now tracked in https://github.com/apache/datasketches-rust/issues/212.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#issuecomment-5402708203
> I can create a couple of stacked PRs for these two?
For sure. I'm kinda a Git guru so any shape that has a good final status should be fine :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#issuecomment-5402890930
Now I'm trying to resolve the major comments on this PR today or tomorrow and merge it so @jeffoodchain or anyone else can work on a basic REQ impl.
We're still in an early development stage so there is no reason to pending a PR for a long time :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814826504
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
+
+ /// Sets the rank accuracy.
+ pub fn rank_accuracy(mut self, ra: RankAccuracy) -> Self {
+ self.rank_accuracy = ra;
+ self
+ }
Review Comment:
```suggestion
pub fn rank_accuracy(mut self, rank_accuracy: RankAccuracy) -> Self {
self.rank_accuracy = rank_accuracy;
self
}
```
nit: public API would prefer explicit over abbr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847061639
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
+
+ /// Sets the rank accuracy.
+ pub fn rank_accuracy(mut self, ra: RankAccuracy) -> Self {
+ self.rank_accuracy = ra;
+ self
+ }
Review Comment:
Have fixed this here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#issuecomment-5400788908
I can create a couple of stacked PRs for these two?
> [CODEX COMMENT STARTS] I found two correctness issues that I believe need to be fixed before this can be merged.
>
> 1. **`Compactor::merge` breaks the ordering invariant required by the wire format**
I had been looking at introducing ordering in merge in a different context and have something working there. This looks like a good catch
>
> 2. **Deserialization accepts impossible compactor states that later panic or violate public API contracts**
Seems reasonable, I'll take a look at this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847125465
##########
datasketches/tests/serde_tests/req.rs:
##########
Review Comment:
moved it here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3814792355
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
Review Comment:
> in the cross-language differences doc.
Seems not included in this PR. We may rewrite this doc comment a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3847029909
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
Review Comment:
Have removed the reference here https://github.com/apache/datasketches-rust/pull/204/changes/d5f9d2f08890160b4372f62ca0fa74d8c27e3296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun merged PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848671819
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
Maybe we can have a build() API that consumes the Builder and returns the final result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848701043
##########
datasketches/src/req/sketch.rs:
##########
@@ -0,0 +1,833 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ sketch — generic over `T: ReqValue`.
+
+use std::fmt;
+
+use super::DEFAULT_K;
+use super::MAX_K;
+use super::MIN_K;
+use super::RankAccuracy;
+use super::SearchCriteria;
+use super::compactor::Compactor;
+use super::iter::ReqSketchIterator;
+use super::sorted_view::SortedView;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// A Relative Error Quantiles sketch for approximate quantile estimation.
+///
+/// See the [module-level documentation](super) for background.
+#[derive(Debug, Clone)]
+pub struct ReqSketch<T: ReqValue> {
+ pub(super) k: u16,
+ pub(super) rank_accuracy: RankAccuracy,
+ pub(super) n: u64,
+ pub(super) max_nom_size: u32,
+ pub(super) num_retained: u32,
+ pub(super) compactors: Vec<Compactor<T>>,
+ pub(super) promotion_buf: Vec<T>,
+ pub(super) min_item: Option<T>,
+ pub(super) max_item: Option<T>,
+}
+
+impl<T: ReqValue> ReqSketch<T> {
+ /// Creates a new sketch with default parameters (`k = 12`, `RankAccuracy::HighRank`).
+ pub fn new() -> Self {
+ let mut s = Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(DEFAULT_K as usize),
+ min_item: None,
+ max_item: None,
+ };
+ // C++ parity: an empty sketch has a level-0 compactor present from the start.
+ // This makes is_raw_items() and flags_byte() byte-compatible with the C++/Java
+ // wire format for the empty case.
+ s.grow();
+ s
+ }
+
+ /// Creates a new sketch with the given `k` and rank accuracy.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn try_new(k: u16, rank_accuracy: RankAccuracy) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ let mut s = Self {
+ k,
+ rank_accuracy,
+ n: 0,
+ max_nom_size: 0,
+ num_retained: 0,
+ compactors: Vec::new(),
+ promotion_buf: Vec::with_capacity(k as usize),
+ min_item: None,
+ max_item: None,
+ };
+ s.grow();
+ Ok(s)
+ }
+
+ /// Returns a builder for chained configuration.
+ pub fn builder() -> ReqSketchBuilder<T> {
+ ReqSketchBuilder::new()
+ }
+
+ /// Returns the configured `k` parameter.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns the configured rank accuracy.
+ pub fn rank_accuracy(&self) -> RankAccuracy {
+ self.rank_accuracy
+ }
+
+ /// Returns the total number of items observed (matches C++ `get_n`).
+ pub fn n(&self) -> u64 {
+ self.n
+ }
+
+ /// Returns true if the sketch has observed no items.
+ pub fn is_empty(&self) -> bool {
+ self.n == 0
+ }
+
+ /// Returns true if compaction has occurred.
+ pub fn is_estimation_mode(&self) -> bool {
+ self.compactors.len() > 1
+ }
+
+ /// Returns the number of items currently stored across all compactors.
+ pub fn num_retained(&self) -> u32 {
+ self.num_retained
+ }
+
+ /// Returns the smallest item ever observed, or `None` if empty.
+ pub fn min_item(&self) -> Option<&T> {
+ self.min_item.as_ref()
+ }
+
+ /// Returns the largest item ever observed, or `None` if empty.
+ pub fn max_item(&self) -> Option<&T> {
+ self.max_item.as_ref()
+ }
+
+ /// Updates the sketch with a new item.
+ ///
+ /// NaN inputs are silently ignored for floating-point types, matching the behavior
+ /// of the Java reference implementation (`checkNaNUpdate`). This is intentional and
+ /// documented in the cross-language differences doc.
+ pub fn update(&mut self, item: T) {
+ if item.is_nan() {
+ return;
+ }
+ match &mut self.min_item {
+ None => self.min_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_lt() => *cur = item.clone(),
+ _ => {}
+ }
+ match &mut self.max_item {
+ None => self.max_item = Some(item.clone()),
+ Some(cur) if item.total_cmp(cur).is_gt() => *cur = item.clone(),
+ _ => {}
+ }
+
+ self.compactors[0].append(item);
+ self.n += 1;
+ self.num_retained += 1;
+
+ if self.num_retained == self.max_nom_size {
+ self.compress();
+ }
+ }
+
+ /// Resets the sketch to the empty state.
+ pub fn reset(&mut self) {
+ self.n = 0;
+ self.num_retained = 0;
+ self.max_nom_size = 0;
+ self.min_item = None;
+ self.max_item = None;
+ self.compactors.clear();
+ self.grow();
+ }
+
+ /// Returns an iterator over `(item, weight)` pairs.
+ pub fn iter(&self) -> ReqSketchIterator<'_, T> {
+ ReqSketchIterator::new(&self.compactors)
+ }
+
+ /// Returns the approximate rank of `item` in `[0.0, 1.0]`.
+ ///
+ /// Computed directly from the retained items in a single `O(retained)` pass,
+ /// without building a sorted view. The result is identical to
+ /// [`SortedView::rank`] on [`Self::sorted_view`].
+ ///
+ /// # Errors
+ /// Returns an error if the sketch is empty or `item` is NaN.
+ pub fn rank(&self, item: &T, criteria: SearchCriteria) -> Result<f64, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if item.is_nan() {
+ return Err(Error::invalid_argument("query item is NaN"));
+ }
+ let inclusive = matches!(criteria, SearchCriteria::Inclusive);
+ let weight: u64 = self
+ .compactors
+ .iter()
+ .map(|c| c.count_below(item, inclusive) as u64 * c.weight())
+ .sum();
+ Ok(weight as f64 / self.n as f64)
+ }
+
+ /// Returns the approximate quantile at the given normalized rank.
+ ///
+ /// Builds a transient [`SortedView`] internally. For repeated quantile
+ /// queries, take one snapshot with [`Self::sorted_view`] and query it.
+ pub fn quantile(&self, rank: f64, criteria: SearchCriteria) -> Result<T, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ if !(0.0..=1.0).contains(&rank) {
+ return Err(Error::invalid_argument(format!(
+ "rank {rank} must be in [0, 1]"
+ )));
+ }
+ self.sorted_view().quantile(rank, criteria)
+ }
+
+ /// Returns approximate quantiles for the given normalized ranks.
+ ///
+ /// The sorted view is built once and shared across all ranks.
+ pub fn quantiles(&self, ranks: &[f64], criteria: SearchCriteria) -> Result<Vec<T>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ // Reject invalid ranks before paying for the view build.
+ for &r in ranks {
+ if !(0.0..=1.0).contains(&r) {
+ return Err(Error::invalid_argument(format!(
+ "rank {r} must be in [0, 1]"
+ )));
+ }
+ }
+ let view = self.sorted_view();
+ ranks.iter().map(|&r| view.quantile(r, criteria)).collect()
+ }
+
+ /// Returns the Probability Mass Function over the given split points.
+ pub fn pmf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().pmf(split_points, criteria)
+ }
+
+ /// Returns the Cumulative Distribution Function over the given split points.
+ pub fn cdf(&self, split_points: &[T], criteria: SearchCriteria) -> Result<Vec<f64>, Error> {
+ if self.is_empty() {
+ return Err(Error::invalid_argument("sketch is empty"));
+ }
+ self.sorted_view().cdf(split_points, criteria)
+ }
+
+ /// Returns an owned, sorted snapshot of the sketch's current state.
+ ///
+ /// An empty sketch yields an empty view; queries on it return an error. The
+ /// view is independent of the sketch — it can be queried (and sent to other
+ /// threads) while the sketch keeps receiving updates, and it keeps answering
+ /// from the state it was taken at.
+ ///
+ /// Building the view costs `O(retained · log retained)`; each query on it is
+ /// then `O(log retained)`. Prefer taking one view for repeated queries over
+ /// calling [`Self::quantile`]/[`Self::pmf`]/[`Self::cdf`], which each build a
+ /// transient view.
+ pub fn sorted_view(&self) -> SortedView<T> {
+ let mut weighted_items = Vec::with_capacity(self.num_retained as usize);
+ for compactor in &self.compactors {
+ let weight = compactor.weight();
+ for item in compactor.iter() {
+ weighted_items.push((item.clone(), weight));
+ }
+ }
+ SortedView::new(weighted_items)
+ }
+
+ /// Merges another sketch into this one.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the two sketches have different `rank_accuracy`.
+ pub fn merge(&mut self, other: &Self) -> Result<(), Error> {
+ if self.rank_accuracy != other.rank_accuracy {
+ return Err(Error::invalid_argument(
+ "sketches must have the same rank_accuracy",
+ ));
+ }
+
+ if other.is_empty() {
+ return Ok(());
+ }
+
+ self.n += other.n;
+
+ if let Some(m) = &other.min_item {
+ match &self.min_item {
+ None => self.min_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_lt() => self.min_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+ if let Some(m) = &other.max_item {
+ match &self.max_item {
+ None => self.max_item = Some(m.clone()),
+ Some(cur) if m.total_cmp(cur).is_gt() => self.max_item = Some(m.clone()),
+ _ => {}
+ }
+ }
+
+ while self.compactors.len() < other.compactors.len() {
+ self.grow();
+ }
+
+ for (i, other_c) in other.compactors.iter().enumerate() {
+ self.compactors[i].merge(other_c);
+ }
+
+ self.update_max_nom_size();
+ self.update_num_retained();
+
+ if self.num_retained >= self.max_nom_size {
+ self.compress();
+ }
+
+ Ok(())
+ }
+
+ /// Returns the lower bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_lower_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_lower_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ /// Returns the upper bound for the rank of a quantile at `num_std_dev` confidence.
+ pub fn rank_upper_bound(&self, rank: f64, num_std_dev: u8) -> f64 {
+ self.compute_rank_upper_bound(
+ self.k,
+ self.compactors.len() as u8,
+ rank,
+ num_std_dev,
+ self.n,
+ matches!(self.rank_accuracy, RankAccuracy::HighRank),
+ )
+ }
+
+ const FIXED_RSE_FACTOR: f64 = 0.084;
+ const INIT_NUM_SECTIONS: u8 = 3;
+
+ fn relative_rse_factor() -> f64 {
+ (0.0512 / Self::INIT_NUM_SECTIONS as f64).sqrt()
+ }
+
+ fn compute_rank_lower_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let lb_rel = rank - num_std_dev as f64 * relative;
+ let lb_fix = rank - num_std_dev as f64 * fixed;
+ lb_rel.max(lb_fix).max(0.0)
+ }
+
+ fn compute_rank_upper_bound(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ num_std_dev: u8,
+ n: u64,
+ hra: bool,
+ ) -> f64 {
+ if self.is_exact_rank_threshold(k, num_levels, rank, n, hra) {
+ return rank;
+ }
+ let relative = Self::relative_rse_factor() / k as f64 * if hra { 1.0 - rank } else { rank };
+ let fixed = Self::FIXED_RSE_FACTOR / k as f64;
+ let ub_rel = rank + num_std_dev as f64 * relative;
+ let ub_fix = rank + num_std_dev as f64 * fixed;
+ ub_rel.min(ub_fix).min(1.0)
+ }
+
+ fn is_exact_rank_threshold(
+ &self,
+ k: u16,
+ num_levels: u8,
+ rank: f64,
+ n: u64,
+ hra: bool,
+ ) -> bool {
+ let base_cap = k as u64 * Self::INIT_NUM_SECTIONS as u64;
+ if num_levels == 1 || n <= base_cap {
+ return true;
+ }
+ let exact_rank_thresh = base_cap as f64 / n as f64;
+ if hra {
+ rank >= 1.0 - exact_rank_thresh
+ } else {
+ rank <= exact_rank_thresh
+ }
+ }
+
+ /// Returns per-level info: `(level_index, num_items, capacity, weight)`.
+ /// Internal/test API; subject to change.
+ #[doc(hidden)]
+ pub fn level_info(&self) -> Vec<(usize, u32, u32, u64)> {
+ self.compactors
+ .iter()
+ .enumerate()
+ .map(|(i, c)| (i, c.num_items(), c.nominal_capacity(), c.weight()))
+ .collect()
+ }
+
+ /// Total nominal capacity across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_nominal_capacity(&self) -> u32 {
+ self.compactors.iter().map(|c| c.nominal_capacity()).sum()
+ }
+
+ /// Total retained items across all levels. Internal/test API.
+ #[doc(hidden)]
+ pub fn total_retained_items(&self) -> u32 {
+ self.compactors.iter().map(|c| c.num_items()).sum()
+ }
+
+ /// Sum of `level_items × level_weight` across compactors. Internal/test API.
+ #[doc(hidden)]
+ pub fn computed_total_weight(&self) -> u64 {
+ self.compactors
+ .iter()
+ .map(|c| c.num_items() as u64 * c.weight())
+ .sum()
+ }
+
+ pub(super) fn flags_byte(&self) -> u8 {
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ let mut flags = 0u8;
+ if self.is_empty() {
+ flags |= FLAG_IS_EMPTY;
+ }
+ if matches!(self.rank_accuracy, RankAccuracy::HighRank) {
+ flags |= FLAG_IS_HIGH_RANK;
+ }
+ if self.is_raw_items() {
+ flags |= FLAG_RAW_ITEMS;
+ }
+ if self.compactors[0].is_sorted() {
+ flags |= FLAG_IS_LEVEL_ZERO_SORTED;
+ }
+ flags
+ }
+
+ pub(super) fn is_raw_items(&self) -> bool {
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ self.n <= RAW_ITEMS_THRESHOLD && self.compactors.len() == 1
+ }
+
+ /// Number of bytes required to serialize the sketch.
+ pub fn serialized_size_bytes(&self) -> usize {
+ // Fixed sketch preamble: 8 bytes (preamble_ints, serial_version, family,
+ // flags, k(2), num_levels, num_raw_items).
+ let mut size = 8usize;
+ if self.is_empty() {
+ return size;
+ }
+ if self.is_estimation_mode() {
+ size += 8; // n
+ size += T::serialize_size(self.min_item.as_ref().unwrap());
+ size += T::serialize_size(self.max_item.as_ref().unwrap());
+ }
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ size += T::serialize_size(item);
+ }
+ } else {
+ for c in &self.compactors {
+ // 20-byte compactor preamble + items
+ size += 20;
+ for item in c.iter() {
+ size += T::serialize_size(item);
+ }
+ }
+ }
+ size
+ }
+
+ /// Serialize the sketch into a `Vec<u8>` matching the C++/Java REQ wire format.
+ pub fn serialize(&self) -> Vec<u8> {
+ use super::serialization::PREAMBLE_INTS_ESTIMATION;
+ use super::serialization::PREAMBLE_INTS_EXACT;
+ use super::serialization::SERIAL_VERSION;
+ use crate::codec::SketchBytes;
+ use crate::codec::family::Family;
+
+ let mut out = SketchBytes::with_capacity(self.serialized_size_bytes());
+ let preamble_ints = if self.is_estimation_mode() {
+ PREAMBLE_INTS_ESTIMATION
+ } else {
+ PREAMBLE_INTS_EXACT
+ };
+ out.write_u8(preamble_ints);
+ out.write_u8(SERIAL_VERSION);
+ out.write_u8(Family::REQ.id);
+ out.write_u8(self.flags_byte());
+ out.write_u16_le(self.k);
+ let num_levels = if self.is_empty() {
+ 0
+ } else {
+ self.compactors.len() as u8
+ };
+ out.write_u8(num_levels);
+ let num_raw_items = if self.is_raw_items() { self.n as u8 } else { 0 };
+ out.write_u8(num_raw_items);
+
+ if self.is_empty() {
+ return out.into_bytes();
+ }
+
+ if self.is_estimation_mode() {
+ out.write_u64_le(self.n);
+ self.min_item.as_ref().unwrap().serialize_value(&mut out);
+ self.max_item.as_ref().unwrap().serialize_value(&mut out);
+ }
+
+ if self.is_raw_items() {
+ for item in self.compactors[0].iter() {
+ item.serialize_value(&mut out);
+ }
+ } else {
+ for c in &self.compactors {
+ c.serialize_into(&mut out);
+ }
+ }
+
+ out.into_bytes()
+ }
+
+ /// Deserialize a sketch from bytes produced by [`Self::serialize`] or by the
+ /// C++/Java reference implementations.
+ pub fn deserialize(bytes: &[u8]) -> Result<Self, Error> {
+ use super::compactor::Compactor;
+ use super::serialization::FLAG_IS_EMPTY;
+ use super::serialization::FLAG_IS_HIGH_RANK;
+ use super::serialization::FLAG_IS_LEVEL_ZERO_SORTED;
+ use super::serialization::FLAG_RAW_ITEMS;
+ use super::serialization::RAW_ITEMS_THRESHOLD;
+ use super::serialization::check_preamble_ints;
+ use super::serialization::check_serial_version;
+ use crate::codec::SketchSlice;
+ use crate::codec::assert::insufficient_data;
+ use crate::codec::family::Family;
+
+ let mut cursor = SketchSlice::new(bytes);
+ let preamble_ints = cursor
+ .read_u8()
+ .map_err(insufficient_data("preamble_ints"))?;
+ let serial_version = cursor
+ .read_u8()
+ .map_err(insufficient_data("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(insufficient_data("family_id"))?;
+ let flags = cursor.read_u8().map_err(insufficient_data("flags"))?;
+ let k = cursor.read_u16_le().map_err(insufficient_data("k"))?;
+ let num_levels = cursor.read_u8().map_err(insufficient_data("num_levels"))?;
+ let num_raw_items = cursor
+ .read_u8()
+ .map_err(insufficient_data("num_raw_items"))?;
+
+ check_preamble_ints(preamble_ints, num_levels)?;
+ check_serial_version(serial_version)?;
+ Family::REQ.validate_id(family_id)?;
+
+ let is_empty = flags & FLAG_IS_EMPTY != 0;
+ let hra = flags & FLAG_IS_HIGH_RANK != 0;
+ let raw_items = flags & FLAG_RAW_ITEMS != 0;
+ let is_level_zero_sorted = flags & FLAG_IS_LEVEL_ZERO_SORTED != 0;
+
+ let rank_accuracy = if hra {
+ RankAccuracy::HighRank
+ } else {
+ RankAccuracy::LowRank
+ };
+ if !(MIN_K..=MAX_K).contains(&k) || k % 2 != 0 {
+ return Err(Error::invalid_argument(format!(
+ "k {k} is not a valid REQ k value"
+ )));
+ }
+
+ if is_empty {
+ if num_levels != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 levels, got {num_levels}"
+ )));
+ }
+ if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "empty REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+ return ReqSketch::try_new(k, rank_accuracy);
+ }
+
+ if num_levels == 0 {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch must have at least one level",
+ ));
+ }
+
+ if raw_items {
+ if num_levels != 1 {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must have exactly 1 level, got {num_levels}"
+ )));
+ }
+ if num_raw_items == 0 || num_raw_items as u64 > RAW_ITEMS_THRESHOLD {
+ return Err(Error::invalid_argument(format!(
+ "raw-items REQ sketch must contain 1..={RAW_ITEMS_THRESHOLD} items, got {num_raw_items}"
+ )));
+ }
+ } else if num_raw_items != 0 {
+ return Err(Error::invalid_argument(format!(
+ "non-raw REQ sketch must have 0 raw items, got {num_raw_items}"
+ )));
+ }
+
+ let mut min_item: Option<T> = None;
+ let mut max_item: Option<T> = None;
+ let mut n: u64 = 1;
+
+ if num_levels > 1 {
+ n = cursor.read_u64_le().map_err(insufficient_data("n"))?;
+ min_item = Some(T::deserialize_value(&mut cursor)?);
+ max_item = Some(T::deserialize_value(&mut cursor)?);
+ }
+
+ let mut compactors: Vec<Compactor<T>> = Vec::with_capacity(num_levels as usize);
+
+ if raw_items {
+ // Single compactor at level 0; items follow directly.
+ let mut items = Vec::with_capacity(num_raw_items as usize);
+ for _ in 0..num_raw_items {
+ items.push(T::deserialize_value(&mut cursor)?);
+ }
+ let c =
+ Compactor::<T>::raw_items_compactor(k, rank_accuracy, items, is_level_zero_sorted);
+ compactors.push(c);
+ } else {
+ for i in 0..num_levels {
+ let level_sorted = if i == 0 { is_level_zero_sorted } else { true };
+ let c = Compactor::<T>::deserialize(&mut cursor, rank_accuracy, level_sorted)?;
+ compactors.push(c);
+ }
+ }
+
+ if num_levels == 1 {
+ // Recover n / min / max from level 0 (these aren't in the preamble for exact mode).
+ let level0 = &compactors[0];
+ n = level0.num_items() as u64;
+ let mut iter = level0.iter();
+ if let Some(first) = iter.next() {
+ let mut mn = first.clone();
+ let mut mx = first.clone();
+ for x in iter {
+ if x.total_cmp(&mn).is_lt() {
+ mn = x.clone();
+ }
+ if x.total_cmp(&mx).is_gt() {
+ mx = x.clone();
+ }
+ }
+ min_item = Some(mn);
+ max_item = Some(mx);
+ }
+ }
+
+ if n == 0 || min_item.is_none() || max_item.is_none() {
+ return Err(Error::invalid_argument(
+ "non-empty REQ sketch contains no items",
+ ));
+ }
+
+ let mut sketch = ReqSketch::try_new(k, rank_accuracy)?;
+ sketch.n = n;
+ sketch.min_item = min_item;
+ sketch.max_item = max_item;
+ sketch.compactors = compactors;
+ sketch.update_max_nom_size();
+ sketch.update_num_retained();
+ Ok(sketch)
+ }
+
+ // --- Internal ---
+
+ pub(super) fn grow(&mut self) {
+ let level = self.compactors.len() as u8;
+ let compactor = Compactor::new(level, self.k, self.rank_accuracy);
+ self.compactors.push(compactor);
+ self.update_max_nom_size();
+ }
+
+ pub(super) fn compress(&mut self) {
+ for h in 0..self.compactors.len() {
+ if self.compactors[h].num_items() >= self.compactors[h].nominal_capacity() {
+ if h == 0 {
+ self.compactors[0].sort();
+ }
+ if h + 1 >= self.compactors.len() {
+ self.grow();
+ }
+ self.promotion_buf.clear();
+ self.compactors[h].compact_into(self.rank_accuracy, &mut self.promotion_buf);
+ if !self.promotion_buf.is_empty() {
+ self.compactors[h + 1].sort();
+ self.compactors[h + 1].merge_sorted(&self.promotion_buf);
+ }
+ self.update_max_nom_size();
+ self.update_num_retained();
+ }
+ }
+ }
+
+ pub(super) fn update_max_nom_size(&mut self) {
+ self.max_nom_size = self.compactors.iter().map(|c| c.nominal_capacity()).sum();
+ }
+
+ pub(super) fn update_num_retained(&mut self) {
+ self.num_retained = self.compactors.iter().map(|c| c.num_items()).sum();
+ }
+}
+
+impl<T: ReqValue> Default for ReqSketch<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+/// Builder for [`ReqSketch`].
+#[derive(Debug, Clone)]
+pub struct ReqSketchBuilder<T: ReqValue> {
+ k: u16,
+ rank_accuracy: RankAccuracy,
+ _marker: std::marker::PhantomData<T>,
+}
+
+impl<T: ReqValue> Default for ReqSketchBuilder<T> {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ReqValue> ReqSketchBuilder<T> {
+ /// Creates a new builder with default parameters.
+ pub fn new() -> Self {
+ Self {
+ k: DEFAULT_K,
+ rank_accuracy: RankAccuracy::HighRank,
+ _marker: std::marker::PhantomData,
+ }
+ }
+
+ /// Sets the `k` parameter.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if `k` is odd or outside `[MIN_K, MAX_K]`.
+ pub fn k(mut self, k: u16) -> Result<Self, Error> {
+ if !(MIN_K..=MAX_K).contains(&k) {
+ return Err(Error::invalid_argument(format!(
+ "k must be in [{}, {}], got {k}",
+ MIN_K, MAX_K
+ )));
+ }
+ if k % 2 != 0 {
+ return Err(Error::invalid_argument(format!("k must be even, got {k}")));
+ }
+ self.k = k;
+ Ok(self)
+ }
Review Comment:
I can help with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3848712368
##########
datasketches/src/req/union.rs:
##########
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! REQ union — combines REQ sketches into a single result.
+
+use super::RankAccuracy;
+use super::sketch::ReqSketch;
+use super::value::ReqValue;
+use crate::error::Error;
+
+/// Combines multiple [`ReqSketch`]es into a single result.
+///
+/// Internally wraps a `ReqSketch` configured for union semantics.
Review Comment:
```suggestion
/// Internally wraps a `ReqSketch` configured for union semantics.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "pmcgleenon (via GitHub)" <gi...@apache.org>.
pmcgleenon commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3824259755
##########
datasketches/Cargo.toml:
##########
@@ -87,9 +93,14 @@ name = "tuple_test"
path = "tests/tuple_test/main.rs"
required-features = ["tuple"]
+[dependencies]
+rand = { workspace = true, optional = true }
Review Comment:
[fastrand](https://crates.io/crates/fastrand) might also be a good option. I saw some benchmarks [here](https://github.com/hoxxep/rapidrand#performance)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat(req): port of reqsketch-rs (datasketches-rust)
Posted by "jeffoodchain (via GitHub)" <gi...@apache.org>.
jeffoodchain commented on code in PR #204:
URL: https://github.com/apache/datasketches-rust/pull/204#discussion_r3821229015
##########
datasketches/src/req/compactor.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Compactor implementation for REQ sketch levels.
+//!
+//! Each level in the REQ sketch uses a compactor to maintain a bounded set of items
+//! with deterministic compaction when capacity is exceeded.
+
+use super::RankAccuracy;
+use super::value::ReqValue;
+use crate::error::Error;
+
+fn nearest_even(value: f32) -> u32 {
+ ((value / 2.0).round() as u32) << 1
+}
+
+/// A compactor maintains items at a specific level of the REQ sketch.
+///
+/// When the compactor reaches its nominal capacity, it performs compaction
+/// by keeping approximately half the items and promoting the rest to the next level.
+#[derive(Debug, Clone)]
+pub(super) struct Compactor<T> {
+ /// Current items in the compactor
+ items: Vec<T>,
+ /// Whether items are currently sorted
+ is_sorted: bool,
+ /// State for deterministic compaction
+ state: u64,
+ /// Reusable scratch buffer for compaction operations
+ scratch_buffer: Vec<T>,
+
+ /// Actual section size (rounded to integer)
+ section_size: u32,
+ /// Number of sections in this compactor
+ num_sections: u8,
+ /// The level of this compactor (0 = base level)
+ lg_weight: u8,
+
+ /// Whether this compactor is configured for high rank accuracy
+ rank_accuracy: RankAccuracy,
+ /// Raw section size (may be fractional)
+ section_size_raw: f32,
+ /// Random bit for compaction
+ coin: bool,
+}
+
+impl<T> Compactor<T>
+where
+ T: Clone + ReqValue,
+{
+ /// Creates a new compactor for the given level.
+ ///
+ /// # Arguments
+ /// * `lg_weight` - The level (log weight) of this compactor
+ /// * `k` - The k parameter from the parent sketch
+ /// * `rank_accuracy` - Rank accuracy configuration
+ pub(super) fn new(lg_weight: u8, k: u16, rank_accuracy: RankAccuracy) -> Self {
+ let section_size_raw = k as f32;
+ let section_size = nearest_even(section_size_raw);
+ let num_sections = 3u8;
+
+ let nominal: usize = (2 * section_size * num_sections as u32) as usize;
+
+ Self {
+ items: Vec::with_capacity(nominal),
+ is_sorted: true,
+ state: 0,
+ scratch_buffer: Vec::with_capacity(nominal / 2 + 8),
+
+ section_size,
+ num_sections,
+ lg_weight,
+
+ rank_accuracy,
+ section_size_raw,
+ coin: false,
+ }
+ }
+
+ /// Returns the number of items currently in this compactor.
+ pub(super) fn num_items(&self) -> u32 {
+ self.items.len() as u32
+ }
+
+ /// Returns the nominal capacity of this compactor.
+ pub(super) fn nominal_capacity(&self) -> u32 {
+ 2 * self.section_size * self.num_sections as u32
+ }
+
+ /// Returns whether the items are currently sorted.
+ pub(super) fn is_sorted(&self) -> bool {
+ self.is_sorted
+ }
+
+ /// Appends an item to this compactor.
+ #[inline(always)]
+ pub(super) fn append(&mut self, item: T) {
+ self.items.push(item);
+ if self.items.len() > 1 {
+ self.is_sorted = false;
+ }
+ }
+
+ /// Merges items from another compactor into this one.
+ pub(super) fn merge(&mut self, other: &Self) {
+ self.state |= other.state;
+ self.items.extend_from_slice(&other.items);
+ if !other.items.is_empty() {
+ self.is_sorted = false;
+ }
+ // OR-ing the schedule counters can advance state past several doubling
+ // thresholds at once. Loop until no more doublings are needed (C++:
+ // req_compactor_impl.hpp:250 — `while (ensure_enough_sections()) {}`).
+ while self.ensure_enough_sections() {}
+ }
+
+ /// Counts the items at-or-below (`inclusive`) or strictly below `item`.
+ ///
+ /// Uses binary search when this compactor is sorted, and a linear scan
+ /// otherwise. This lets [`ReqSketch::rank`](super::ReqSketch::rank) sum
+ /// per-level weights directly without first building a sorted view.
+ pub(super) fn count_below(&self, item: &T, inclusive: bool) -> usize {
+ if self.is_sorted {
+ if inclusive {
+ self.items.partition_point(|x| x.total_cmp(item).is_le())
+ } else {
+ self.items.partition_point(|x| x.total_cmp(item).is_lt())
+ }
+ } else {
+ self.items
+ .iter()
+ .filter(|x| {
+ let ord = x.total_cmp(item);
+ if inclusive { ord.is_le() } else { ord.is_lt() }
+ })
+ .count()
+ }
+ }
+
+ /// Merges pre-sorted items into this compactor.
+ /// Merges sorted items into this compactor using scratch buffer to avoid allocation.
+ /// Both this compactor's items and the input must be sorted.
+ #[inline(always)]
+ pub(super) fn merge_sorted(&mut self, items: &[T]) {
+ if items.is_empty() {
+ return;
+ }
+
+ if self.items.is_empty() {
+ self.items.extend_from_slice(items);
+ self.is_sorted = true;
+ return;
+ }
+
+ // Ensure sorted on both inputs by contract
+ let total = self.items.len() + items.len();
+ self.scratch_buffer.clear();
+ if self.scratch_buffer.capacity() < total {
+ self.scratch_buffer
+ .reserve(total - self.scratch_buffer.capacity());
+ }
+
+ let (mut i, mut j) = (0usize, 0usize);
+ let (a, b) = (&self.items, items);
+
+ // Two-pointer merge into scratch buffer
+ while i < a.len() && j < b.len() {
+ if a[i].total_cmp(&b[j]).is_le() {
+ self.scratch_buffer.push(a[i].clone());
+ i += 1;
+ } else {
+ self.scratch_buffer.push(b[j].clone());
+ j += 1;
+ }
+ }
+
+ // Add remaining elements
+ if i < a.len() {
+ self.scratch_buffer.extend_from_slice(&a[i..]);
+ }
+ if j < b.len() {
+ self.scratch_buffer.extend_from_slice(&b[j..]);
+ }
+
+ // Swap scratch buffer with items (zero-copy)
+ self.items.clear();
+ std::mem::swap(&mut self.items, &mut self.scratch_buffer);
+ self.is_sorted = true;
+ }
+
+ /// Sorts the items in this compactor if not already sorted.
+ #[inline(always)]
+ pub(super) fn sort(&mut self) {
+ if !self.is_sorted {
+ // Use unstable sort for better performance (stable not needed for REQ sketch)
+ self.items.sort_unstable_by(|a, b| a.total_cmp(b));
+ self.is_sorted = true;
+ }
+ }
+
+ /// Compacts into the provided output buffer without allocating.
+ /// Writes promoted items into `out` and removes the compacted range in-place via `copy_within +
+ /// truncate`.
+ #[inline(always)]
+ pub(super) fn compact_into(&mut self, _rank_accuracy: RankAccuracy, out: &mut Vec<T>) {
+ if self.items.is_empty() {
+ out.clear();
+ return;
+ }
+
+ // Sort entire buffer (C++ sorts full buffer before compaction)
+ self.sort();
+
+ // Calculate sections to compact based on state
+ let secs_to_compact =
+ ((!self.state).trailing_zeros() + 1).min(self.num_sections as u32) as u8;
+ let compaction_range = self.compute_compaction_range(secs_to_compact);
+
+ // Must have at least 2 items to compact
+ if compaction_range.1 <= compaction_range.0 || (compaction_range.1 - compaction_range.0) < 2
+ {
+ out.clear();
+ return;
+ }
+
+ if (self.state & 1) == 1 {
+ self.coin = !self.coin; // flip coin for odd states
+ } else {
+ self.coin = rand::random::<bool>(); // random coin flip for even states
+ }
+ let odds = self.coin;
+
+ // Build promoted items directly into output buffer (no alloc)
+ out.clear();
+ let (start, end) = compaction_range;
+ let mut i = start + if odds { 1 } else { 0 };
+ while i < end {
+ out.push(self.items[i].clone()); // TODO: use Copy fast-path for numeric types
+ i += 2;
+ }
+
+ // Remove the compacted range in-place by rotating elements left
+ let removed = end - start;
+ if end < self.items.len() {
+ // Use rotate_left to move tail elements to fill the gap
+ self.items[start..].rotate_left(removed);
+ }
+ self.items.truncate(self.items.len() - removed);
+
+ // Update state, then ensure enough sections (C++ order)
+ self.state += 1;
+ self.ensure_enough_sections();
+ }
+
+ /// Returns an iterator over the items in this compactor.
+ pub(super) fn iter(&self) -> impl Iterator<Item = &T> {
+ self.items.iter()
+ }
+
+ /// Returns a slice of items for zero-allocation iteration.
+ pub(super) fn items_slice(&self) -> &[T] {
+ &self.items
+ }
+
+ /// Returns the weight (2^lg_weight) for items in this compactor.
+ pub(super) fn weight(&self) -> u64 {
+ 1u64 << self.lg_weight
+ }
+
+ // Private helper methods
+
+ fn ensure_enough_sections(&mut self) -> bool {
+ let ssr = self.section_size_raw / (2.0_f32).sqrt();
Review Comment:
misc: `std::f32::consts::SQRT_2` might be a great idea here.
##########
datasketches/src/req/compactor.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Compactor implementation for REQ sketch levels.
+//!
+//! Each level in the REQ sketch uses a compactor to maintain a bounded set of items
+//! with deterministic compaction when capacity is exceeded.
+
+use super::RankAccuracy;
+use super::value::ReqValue;
+use crate::error::Error;
+
+fn nearest_even(value: f32) -> u32 {
+ ((value / 2.0).round() as u32) << 1
+}
+
+/// A compactor maintains items at a specific level of the REQ sketch.
+///
+/// When the compactor reaches its nominal capacity, it performs compaction
+/// by keeping approximately half the items and promoting the rest to the next level.
+#[derive(Debug, Clone)]
+pub(super) struct Compactor<T> {
+ /// Current items in the compactor
+ items: Vec<T>,
+ /// Whether items are currently sorted
+ is_sorted: bool,
+ /// State for deterministic compaction
+ state: u64,
+ /// Reusable scratch buffer for compaction operations
+ scratch_buffer: Vec<T>,
+
+ /// Actual section size (rounded to integer)
+ section_size: u32,
+ /// Number of sections in this compactor
+ num_sections: u8,
+ /// The level of this compactor (0 = base level)
+ lg_weight: u8,
+
+ /// Whether this compactor is configured for high rank accuracy
+ rank_accuracy: RankAccuracy,
+ /// Raw section size (may be fractional)
+ section_size_raw: f32,
+ /// Random bit for compaction
+ coin: bool,
+}
+
+impl<T> Compactor<T>
+where
+ T: Clone + ReqValue,
+{
+ /// Creates a new compactor for the given level.
+ ///
+ /// # Arguments
+ /// * `lg_weight` - The level (log weight) of this compactor
+ /// * `k` - The k parameter from the parent sketch
+ /// * `rank_accuracy` - Rank accuracy configuration
+ pub(super) fn new(lg_weight: u8, k: u16, rank_accuracy: RankAccuracy) -> Self {
+ let section_size_raw = k as f32;
+ let section_size = nearest_even(section_size_raw);
+ let num_sections = 3u8;
+
+ let nominal: usize = (2 * section_size * num_sections as u32) as usize;
+
+ Self {
+ items: Vec::with_capacity(nominal),
+ is_sorted: true,
+ state: 0,
+ scratch_buffer: Vec::with_capacity(nominal / 2 + 8),
+
+ section_size,
+ num_sections,
+ lg_weight,
+
+ rank_accuracy,
+ section_size_raw,
+ coin: false,
+ }
+ }
+
+ /// Returns the number of items currently in this compactor.
+ pub(super) fn num_items(&self) -> u32 {
+ self.items.len() as u32
+ }
+
+ /// Returns the nominal capacity of this compactor.
+ pub(super) fn nominal_capacity(&self) -> u32 {
+ 2 * self.section_size * self.num_sections as u32
+ }
+
+ /// Returns whether the items are currently sorted.
+ pub(super) fn is_sorted(&self) -> bool {
+ self.is_sorted
+ }
+
+ /// Appends an item to this compactor.
+ #[inline(always)]
+ pub(super) fn append(&mut self, item: T) {
+ self.items.push(item);
+ if self.items.len() > 1 {
+ self.is_sorted = false;
+ }
+ }
+
+ /// Merges items from another compactor into this one.
+ pub(super) fn merge(&mut self, other: &Self) {
+ self.state |= other.state;
+ self.items.extend_from_slice(&other.items);
+ if !other.items.is_empty() {
+ self.is_sorted = false;
+ }
+ // OR-ing the schedule counters can advance state past several doubling
+ // thresholds at once. Loop until no more doublings are needed (C++:
+ // req_compactor_impl.hpp:250 — `while (ensure_enough_sections()) {}`).
+ while self.ensure_enough_sections() {}
+ }
+
+ /// Counts the items at-or-below (`inclusive`) or strictly below `item`.
+ ///
+ /// Uses binary search when this compactor is sorted, and a linear scan
+ /// otherwise. This lets [`ReqSketch::rank`](super::ReqSketch::rank) sum
+ /// per-level weights directly without first building a sorted view.
+ pub(super) fn count_below(&self, item: &T, inclusive: bool) -> usize {
+ if self.is_sorted {
+ if inclusive {
+ self.items.partition_point(|x| x.total_cmp(item).is_le())
+ } else {
+ self.items.partition_point(|x| x.total_cmp(item).is_lt())
+ }
+ } else {
+ self.items
+ .iter()
+ .filter(|x| {
+ let ord = x.total_cmp(item);
+ if inclusive { ord.is_le() } else { ord.is_lt() }
+ })
+ .count()
+ }
+ }
+
+ /// Merges pre-sorted items into this compactor.
+ /// Merges sorted items into this compactor using scratch buffer to avoid allocation.
+ /// Both this compactor's items and the input must be sorted.
+ #[inline(always)]
+ pub(super) fn merge_sorted(&mut self, items: &[T]) {
+ if items.is_empty() {
+ return;
+ }
+
+ if self.items.is_empty() {
+ self.items.extend_from_slice(items);
+ self.is_sorted = true;
+ return;
+ }
+
+ // Ensure sorted on both inputs by contract
+ let total = self.items.len() + items.len();
+ self.scratch_buffer.clear();
+ if self.scratch_buffer.capacity() < total {
+ self.scratch_buffer
+ .reserve(total - self.scratch_buffer.capacity());
+ }
+
+ let (mut i, mut j) = (0usize, 0usize);
+ let (a, b) = (&self.items, items);
+
+ // Two-pointer merge into scratch buffer
+ while i < a.len() && j < b.len() {
+ if a[i].total_cmp(&b[j]).is_le() {
+ self.scratch_buffer.push(a[i].clone());
+ i += 1;
+ } else {
+ self.scratch_buffer.push(b[j].clone());
+ j += 1;
+ }
+ }
+
+ // Add remaining elements
+ if i < a.len() {
+ self.scratch_buffer.extend_from_slice(&a[i..]);
+ }
+ if j < b.len() {
+ self.scratch_buffer.extend_from_slice(&b[j..]);
+ }
+
+ // Swap scratch buffer with items (zero-copy)
+ self.items.clear();
+ std::mem::swap(&mut self.items, &mut self.scratch_buffer);
+ self.is_sorted = true;
+ }
+
+ /// Sorts the items in this compactor if not already sorted.
+ #[inline(always)]
+ pub(super) fn sort(&mut self) {
+ if !self.is_sorted {
+ // Use unstable sort for better performance (stable not needed for REQ sketch)
+ self.items.sort_unstable_by(|a, b| a.total_cmp(b));
+ self.is_sorted = true;
+ }
+ }
+
+ /// Compacts into the provided output buffer without allocating.
+ /// Writes promoted items into `out` and removes the compacted range in-place via `copy_within +
+ /// truncate`.
+ #[inline(always)]
+ pub(super) fn compact_into(&mut self, _rank_accuracy: RankAccuracy, out: &mut Vec<T>) {
+ if self.items.is_empty() {
+ out.clear();
+ return;
+ }
+
+ // Sort entire buffer (C++ sorts full buffer before compaction)
+ self.sort();
+
+ // Calculate sections to compact based on state
+ let secs_to_compact =
+ ((!self.state).trailing_zeros() + 1).min(self.num_sections as u32) as u8;
+ let compaction_range = self.compute_compaction_range(secs_to_compact);
+
+ // Must have at least 2 items to compact
+ if compaction_range.1 <= compaction_range.0 || (compaction_range.1 - compaction_range.0) < 2
+ {
+ out.clear();
+ return;
+ }
+
+ if (self.state & 1) == 1 {
+ self.coin = !self.coin; // flip coin for odd states
+ } else {
+ self.coin = rand::random::<bool>(); // random coin flip for even states
+ }
+ let odds = self.coin;
+
+ // Build promoted items directly into output buffer (no alloc)
+ out.clear();
+ let (start, end) = compaction_range;
+ let mut i = start + if odds { 1 } else { 0 };
+ while i < end {
+ out.push(self.items[i].clone()); // TODO: use Copy fast-path for numeric types
+ i += 2;
+ }
+
+ // Remove the compacted range in-place by rotating elements left
+ let removed = end - start;
+ if end < self.items.len() {
+ // Use rotate_left to move tail elements to fill the gap
+ self.items[start..].rotate_left(removed);
+ }
+ self.items.truncate(self.items.len() - removed);
+
+ // Update state, then ensure enough sections (C++ order)
+ self.state += 1;
+ self.ensure_enough_sections();
+ }
+
+ /// Returns an iterator over the items in this compactor.
+ pub(super) fn iter(&self) -> impl Iterator<Item = &T> {
+ self.items.iter()
+ }
+
+ /// Returns a slice of items for zero-allocation iteration.
+ pub(super) fn items_slice(&self) -> &[T] {
+ &self.items
+ }
+
+ /// Returns the weight (2^lg_weight) for items in this compactor.
+ pub(super) fn weight(&self) -> u64 {
+ 1u64 << self.lg_weight
+ }
+
+ // Private helper methods
+
+ fn ensure_enough_sections(&mut self) -> bool {
+ let ssr = self.section_size_raw / (2.0_f32).sqrt();
+ let ne = nearest_even(ssr);
+
+ const MIN_K: u32 = 4; // matches datasketches-cpp
Review Comment:
I'm curious why not using the one declared in `req/mod.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org