You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "tisonkun (via GitHub)" <gi...@apache.org> on 2025/12/16 05:15:28 UTC
[PR] feat: implement T-Digest (datasketches-rust)
tisonkun opened a new pull request, #23:
URL: https://github.com/apache/datasketches-rust/pull/23
Partially implemented now. I'd finish get_rank today and figure what "CDF" and "PMF" are. Serde later.
This closes #14
Some open questions:
* Shall we support tdigest over f32? Java implements only TDigestDouble while C++ has both for float and double. I think f64 is enough to cover most cases.
* get_rank and get_quantile now takes `&mut self` because they call compress internally. I think we can add a "freezed" tdigest struct that will never updated then and already compressed, so that the freezed struct can be shared anywhere to test ranks and quantities without mut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3662928810
Added cross-language serde compat tests b37f08b
See https://github.com/apache/datasketches-rust/issues/10#issuecomment-3662868794 for how to generate these snapshots. I think we can use that technique and exclude snapshots from VCS as Java/C++ repo does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623730274
##########
src/tdigest/serialization.rs:
##########
@@ -17,6 +41,45 @@ impl TDigest {
});
bytes.push(SERIAL_VERSION);
bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
Review Comment:
Totally get it wrong.
let x = u16::MAX as u64 * 2 + 10;
let y = u32::MAX as u64;
println!("x: {}, y: {}, x < y: {}", x, y, x < y);
Gives: "x: 131080, y: 4294967295, x < y: true".
Never exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622801010
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Seems C++/Java just assumes there is no NaN in either buffer and the internal centroids.
cc @leerho @AlexanderSaydakov - have you tested this kind of input? What is the expected result?
NaN can be produced when adding two centroids. Even if either of them is not NaN, the result of the addition can be NaN (e.g., an internal Inf - Inf).
https://github.com/apache/datasketches-java/blob/37edd8c9898f8f073aaafad7097cb0949eb84c85/src/main/java/org/apache/datasketches/tdigest/TDigestDouble.java#L592-L594
https://github.com/apache/datasketches-cpp/blob/7bb979d3ef8929e235bcd22d67579e1f695f6ecd/tdigest/include/tdigest.hpp#L79-L82
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3661234667
The remaining task in my mind:
* Implement cdf/pmf functions
* Support reading format of the reference implementation and snapshot tests
* Figure out whether to cover NaN edge cases
> > Shall we support tdigest over f32? Java implements only TDigestDouble while C++ has both for float and double. I think f64 is enough to cover most cases.
>
> Is there any way we can have a `TDigest<T: num::Float>`? I think in some cases having less precision for half of the memory is desirable.
I may not include this feature in this PR, because:
1. The Java impl has only TDigestDouble.
2. Generally, f64 is what you need.
3. Switching between f32 and f64 can increase a lot of type tuning issues and perhaps abstractions only for making this work.
That said, I'll try to see if the `weight` field can be a `f64` so we don't need a type config to associate (f64, u64)/(f32, u32) anyway. But the Java/C++ impl does use an integer type for weight, while using f64 for u64 should be lossless since the only assignment here is:
```rust
self.weight = total_weight;
let total_weight = self.weight + other.weight;
```
cc @leerho @AlexanderSaydakov - any reason we must use an integer for `weight`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622380592
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,445 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](super) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: usize,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: usize = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: usize) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k * 2) + fudge;
+
+ let centroids = Vec::with_capacity(centroids_capacity);
+ let buffer = Vec::with_capacity(centroids_capacity * BUFFER_MULTIPLIER);
Review Comment:
Not quite. When deserializing, this may require a reserve.
Use `Vec` for convenience, and as long as the capacity won't exceed, there is no realloc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622380592
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,445 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](super) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: usize,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: usize = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: usize) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k * 2) + fudge;
+
+ let centroids = Vec::with_capacity(centroids_capacity);
+ let buffer = Vec::with_capacity(centroids_capacity * BUFFER_MULTIPLIER);
Review Comment:
Not quite. When deserializing this may requires a reserve.
Use `Vec` for convention and as long as capacity won't exceed, there is no realloc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623719248
##########
src/tdigest/serialization.rs:
##########
@@ -1,14 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE};
+
+use crate::error::SerdeError;
use crate::tdigest::TDigest;
const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
const SERIAL_VERSION: u8 = 1;
const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
impl TDigest {
/// Serializes this TDigest to bytes.
pub fn serialize(&mut self) -> Vec<u8> {
self.compress();
Review Comment:
We should compress here. Postpone type param of serde methods later.
##########
src/tdigest/serialization.rs:
##########
@@ -1,14 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE};
+
+use crate::error::SerdeError;
use crate::tdigest::TDigest;
const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
const SERIAL_VERSION: u8 = 1;
const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
impl TDigest {
/// Serializes this TDigest to bytes.
pub fn serialize(&mut self) -> Vec<u8> {
self.compress();
Review Comment:
We should compress here. Postpone type of param of serde methods later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho merged PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3673372262
> This is overly big because of all the serialized test *.sk files. We don't check those files in as they are created dynamically. That being said, I keep a copy locally (cause I'm lazy and impatient) but even then I do try to update them frequently.
>
> I would request that these *.sk files be removed from this PR.
Prerequisite implemented in https://github.com/apache/datasketches-rust/pull/29. After #29 merged I can rebase this one onto that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625336137
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,1138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{BE, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+/// The default value of K if one is not specified.
+const DEFAULT_K: u16 = 200;
+/// Multiplier for buffer size relative to centroids capacity.
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](super) for more details.
+#[derive(Debug, Clone)]
+pub struct TDigestMut {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigestMut {
+ fn default() -> Self {
+ TDigestMut::new(DEFAULT_K)
+ }
+}
+
+impl TDigestMut {
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigestMut {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given TDigest into this one
+ pub fn merge(&mut self, other: &TDigestMut) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Freezes this TDigest into an immutable one.
+ pub fn freeze(mut self) -> TDigest {
+ self.compress();
+ TDigest {
+ k: self.k,
+ reverse_merge: self.reverse_merge,
+ min: self.min,
+ max: self.max,
+ centroids: self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ fn view(&mut self) -> TDigestView<'_> {
+ self.compress(); // side effect
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().quantile(rank)
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut total_size = 0;
+ if self.is_empty() || self.is_single_value() {
+ // 1 byte preamble
+ // + 1 byte serial version
+ // + 1 byte family
+ // + 2 bytes k
+ // + 1 byte flags
+ // + 2 bytes unused
+ total_size += size_of::<u64>();
+ } else {
+ // all of the above
+ // + 4 bytes num centroids
+ // + 4 bytes num buffered
+ total_size += size_of::<u64>() * 2;
+ }
+ if self.is_empty() {
+ // nothing more
+ } else if self.is_single_value() {
+ // + 8 bytes single value
+ total_size += size_of::<f64>();
+ } else {
+ // + 8 bytes min
+ // + 8 bytes max
+ total_size += size_of::<f64>() * 2;
+ // + (8+8) bytes per centroid
+ total_size += self.centroids.len() * (size_of::<f64>() + size_of::<u64>());
+ }
+
+ let mut bytes = Vec::with_capacity(total_size);
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ bytes.extend_from_slice(&self.k.to_le_bytes());
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ bytes.extend_from_slice(&0u16.to_le_bytes()); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ return bytes;
+ }
+ bytes.extend_from_slice(&(self.centroids.len() as u32).to_le_bytes());
+ bytes.extend_from_slice(&0u32.to_le_bytes()); // unused
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ bytes.extend_from_slice(&self.max.to_le_bytes());
+ for centroid in &self.centroids {
+ bytes.extend_from_slice(¢roid.mean.to_le_bytes());
+ bytes.extend_from_slice(¢roid.weight.to_le_bytes());
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// Supports reading format of the reference implementation (auto-detected) [^2].
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ /// [^2]: <https://github.com/tdunning/t-digest>
+ pub fn deserialize(bytes: &[u8], is_f32: bool) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(tag.to_string())
+ }
+
+ fn check_non_nan(value: f64, tag: &'static str) -> Result<(), SerdeError> {
+ if value.is_nan() {
+ return Err(SerdeError::MalformedData(format!("{tag} cannot be NaN")));
+ }
+ Ok(())
+ }
+
+ fn check_nonzero(value: u64, tag: &'static str) -> Result<(), SerdeError> {
+ if value == 0 {
+ return Err(SerdeError::MalformedData(format!("{tag} cannot be zero")));
+ }
+ Ok(())
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ if preamble_longs == 0 && serial_version == 0 && family_id == 0 {
+ return Self::deserialize_compat(bytes);
+ }
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigestMut::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ check_non_nan(value, "single_value")?;
+ return Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ check_non_nan(min, "min")?;
+ check_non_nan(max, "max")?;
Review Comment:
min & max should have other certia like min <= max and should be consistent with centroid + buffer. But I'd leave it later since the non_nan check is complex enough here. I don't make this PR too unreviewable (
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622938899
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
Review Comment:
I don't know either. This is from Java/C++ impl.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625332149
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Fixes as many as possible in bebd87c.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622256216
##########
src/tdigest/serialization.rs:
##########
@@ -17,6 +41,45 @@ impl TDigest {
});
bytes.push(SERIAL_VERSION);
bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
Review Comment:
This is for `self.buffer`. While in Java impl we always compress and thus skip all for the buffer, the C++ impl may include it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3667543564
> > any reason we must use an integer for weight?
>
> the total weight is the number of input items. weight of a centroid is the number of items this centroid represents.
Yes. I mean, is there some downside if I'm using f64 (double) here? Because to support type generic over f32/f64 like what datasketches-cpp's tdigest<T>, reducing type combo into one generic type would help.
It's quite wordy to implement something like below in Rust:
```cpp
using W = typename std::conditional<std::is_same<T, double>::value, uint64_t, uint32_t>::type;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623857750
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::{Centroid, TDigest};
+
+const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
+const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
+const SERIAL_VERSION: u8 = 1;
+const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
+
+impl TDigest {
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
Review Comment:
Use this signature as is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3663900981
Here is a little [tutorial on CDFs and PMFs](https://datasketches.apache.org/docs/QuantilesStudies/QuantilesStreamAStudy.html).
It is on the website:
- Click on Documentation
- On the left ToC open Sketch Families
- Open Quantiles and Histograms
- Click on Classic Quantiles Sketches
- At the top of the page click on Quantiles StreamA Study
The beauty of using these quantile sketches for histogramming big data is:
- You only scan the raw data once.
- After that, you can query the sketch with a set of points that define the boundaries of your bins. The result is returned to you as two arrays of points. You choose your own plotting tool. If you don't like what you see, you can query the sketch again with a different set of points. Most histogram plotting software requires you to define your bins before you scan the raw data. These sketches allow you to play with your data in nearly real time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3673210670
> I would request that these *.sk files be removed from this PR.
@leerho Sure. We already do that so far, so originally I'm considering to add a followup to build the snapshot in CI with a script as I'm planning in https://github.com/apache/datasketches-rust/issues/10.
But if this can be a blocker, let me try to implement it beforehand :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3673572534
@leerho Rebased and all sk files are removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622387212
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::{Centroid, TDigest};
+
+const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
+const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
+const SERIAL_VERSION: u8 = 1;
+const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
+
+impl TDigest {
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
Review Comment:
Rust does not support override. Not sure if we should have another method like `pub fn deserialize(bytes: &[u8]) -> Result<Self, SerdeError>` and perhaps make this one as `deserialize_with`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622639736
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean < value {
+ std::cmp::Ordering::Less
+ } else {
+ std::cmp::Ordering::Greater
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean > value {
+ std::cmp::Ordering::Greater
+ } else {
+ std::cmp::Ordering::Less
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ pub(super) fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
Review Comment:
It's guarded by the code logics that this vector would take exactly `buffer || centroids` and then iter over.
Trying to add such an assertion sounds tricky: shall we memo the cap here, when we finish the `extend` inner, assert cap's equality? At that point we're sure what is extended and that makes the assertion a bit redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622938899
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
Review Comment:
I don't know either. This is from Java/C++ impl. I guess some expierence value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622938899
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
Review Comment:
I don't know either. This is from Java/C++ impl. I guess some experience value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623713751
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::{Centroid, TDigest};
+
+const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
+const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
+const SERIAL_VERSION: u8 = 1;
+const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
+
+impl TDigest {
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
Review Comment:
Resolved at 53b74eef5f01604f795c81dbd092a81399b1997c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622707371
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean < value {
+ std::cmp::Ordering::Less
+ } else {
+ std::cmp::Ordering::Greater
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean > value {
+ std::cmp::Ordering::Greater
+ } else {
+ std::cmp::Ordering::Less
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ pub(super) fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ pub(super) fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> std::cmp::Ordering {
Review Comment:
I moved all the cmp logics together in 9497d24.
Although there is `f64::total_cmp`, it takes `+0.0 > -0.0` which is different from [`OrderedFloat`](https://docs.rs/ordered-float/latest/ordered_float/struct.OrderedFloat.html) which is more widely used and give an "intuitive" total order.
Perhaps we can use OrderedFloat in TDigest. Let me think of it ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622253574
##########
src/tdigest/serialization.rs:
##########
@@ -1,14 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE};
+
+use crate::error::SerdeError;
use crate::tdigest::TDigest;
const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
const SERIAL_VERSION: u8 = 1;
const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
impl TDigest {
/// Serializes this TDigest to bytes.
pub fn serialize(&mut self) -> Vec<u8> {
self.compress();
Review Comment:
Consider not to do `compress` here to avoid `&mut self`? But `compress` is not an internal method anyway, and it may reduce some final size.
##########
src/tdigest/serialization.rs:
##########
@@ -1,14 +1,38 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE};
+
+use crate::error::SerdeError;
use crate::tdigest::TDigest;
const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
const SERIAL_VERSION: u8 = 1;
const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
impl TDigest {
/// Serializes this TDigest to bytes.
pub fn serialize(&mut self) -> Vec<u8> {
self.compress();
Review Comment:
Consider not to do `compress` here to avoid `&mut self`? But `compress` is an internal method anyway, and it may reduce some final size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622647025
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
Review Comment:
Similarly, the related elements are pushed just below and in the first line of `do_merge`.
But I do consider if we can move `buffer.extend(std::mem::take(&mut self.centroids));` to the caller so that we keep `tmp`'s init in the same place. We already copy `self.buffer` here anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622630330
##########
src/tdigest/mod.rs:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! T-Digest implementation for estimating quantiles and ranks.
+//!
+//! The implementation in this library is based on the MergingDigest described in
+//! [Computing Extremely Accurate Quantiles Using t-Digests][paper] by Ted Dunning and Otmar Ertl.
+//!
+//! The implementation in this library has a few differences from the reference implementation
+//! associated with that paper:
+//!
+//! * Merge does not modify the input
+//! * Deserialization similar to other sketches in this library, although reading the reference
+//! implementation format is supported
+//!
+//! Unlike all other algorithms in the library, t-digest is empirical and has no mathematical
+//! basis for estimating its error and its results are dependent on the input data. However,
+//! for many common data distributions, it can produce excellent results. t-digest also operates
+//! only on numeric data and, unlike the quantiles family algorithms in the library which return
+//! quantile approximations from the input domain, t-digest interpolates values and will hold and
+//! return data points not seen in the input.
+//!
+//! The closest alternative to t-digest in this library is REQ sketch. It prioritizes one chosen
+//! side of the rank domain: either low rank accuracy or high rank accuracy. t-digest (in this
+//! implementation) prioritizes both ends of the rank domain and has lower accuracy towards the
+//! middle of the rank domain (median).
+//!
+//! Measurements show that t-digest is slightly biased (tends to underestimate low ranks and
+//! overestimate high ranks), while still doing very well close to the extremes. The effect seems
+//! to be more pronounced with more input values.
+//!
+//! For more information on the performance characteristics, see the
+//! [Datasketches page on t-digest](https://datasketches.apache.org/docs/tdigest/tdigest.html).
+//!
+//! [paper]: https://arxiv.org/abs/1902.04023
+
+mod serialization;
+mod sketch;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
Review Comment:
With proper encapsulation, it should not be that awkward. Let me try to move these structs back to `sketch.rs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623009160
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
... and the value from deserialized should be verified and dropped if it is NaN. Or reported as malformed data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "freakyzoidberg (via GitHub)" <gi...@apache.org>.
freakyzoidberg commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3659604786
> cc @notfilippo @freakyzoidberg
>
> cc @leerho based on [#2 (comment)](https://github.com/apache/datasketches-rust/issues/2#issuecomment-3657395474), I made this impl as a combination of C++ & Java version. Welcome to give it a review. I'll add serde support today or tomorrow. But I'm still not quite sure what CDF and PMF are. It's possible to convey C++'s impl as is but I wonder a real world use case to understand its definition and usage.
Without the x-serde this is quite hard to confirm if the synopsis are compatible
But that can be done later a the implem progress
Note I am no mathematician - but my humble understanding
### PMF -> Probability Mass Function
It returns the approximate fraction of data points (mass) that fall into specific "bins" or intervals.
You give it an array of split points and it returns an array of mass fraction that sum up to 1.
it's a histogram generator of sort
### CDF -> Cumulative Distribution Function
It returns the approximate fraction of data points that are less than (or equal to) each split point.
CDF ~= running running sum of the PMF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622627729
##########
src/tdigest/mod.rs:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! T-Digest implementation for estimating quantiles and ranks.
+//!
+//! The implementation in this library is based on the MergingDigest described in
+//! [Computing Extremely Accurate Quantiles Using t-Digests][paper] by Ted Dunning and Otmar Ertl.
+//!
+//! The implementation in this library has a few differences from the reference implementation
+//! associated with that paper:
+//!
+//! * Merge does not modify the input
+//! * Deserialization similar to other sketches in this library, although reading the reference
+//! implementation format is supported
+//!
+//! Unlike all other algorithms in the library, t-digest is empirical and has no mathematical
+//! basis for estimating its error and its results are dependent on the input data. However,
+//! for many common data distributions, it can produce excellent results. t-digest also operates
+//! only on numeric data and, unlike the quantiles family algorithms in the library which return
+//! quantile approximations from the input domain, t-digest interpolates values and will hold and
+//! return data points not seen in the input.
+//!
+//! The closest alternative to t-digest in this library is REQ sketch. It prioritizes one chosen
+//! side of the rank domain: either low rank accuracy or high rank accuracy. t-digest (in this
+//! implementation) prioritizes both ends of the rank domain and has lower accuracy towards the
+//! middle of the rank domain (median).
+//!
+//! Measurements show that t-digest is slightly biased (tends to underestimate low ranks and
+//! overestimate high ranks), while still doing very well close to the extremes. The effect seems
+//! to be more pronounced with more input values.
+//!
+//! For more information on the performance characteristics, see the
+//! [Datasketches page on t-digest](https://datasketches.apache.org/docs/tdigest/tdigest.html).
+//!
+//! [paper]: https://arxiv.org/abs/1902.04023
+
+mod serialization;
+mod sketch;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
Review Comment:
It's possible to move this struct to `sketch.rs` and merge `serialization.rs` into `sketch.rs`. Otherwise, we need to expose the field to `pub(super)` or add several internal methods for the same purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3664280065
> i would reject all +/-Inf on the input, they really mess up data analysis IMHO.
Good point to follow. Let me try to push a commit for doing so later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3674255366
Planning to take a look at this tomorrow. Thanks for your work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2629009194
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,1167 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+use byteorder::{BE, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+use std::num::NonZeroU64;
+
+/// The default value of K if one is not specified.
+const DEFAULT_K: u16 = 200;
+/// Multiplier for buffer size relative to centroids capacity.
+const BUFFER_MULTIPLIER: usize = 4;
+/// Default weight for single values.
+const DEFAULT_WEIGHT: NonZeroU64 = NonZeroU64::new(1).unwrap();
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [tdigest module level documentation](crate::tdigest) for more.
+#[derive(Debug, Clone)]
+pub struct TDigestMut {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigestMut {
+ fn default() -> Self {
+ TDigestMut::new(DEFAULT_K)
+ }
+}
+
+impl TDigestMut {
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigestMut {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value.
+ ///
+ /// [f64::NAN], [f64::INFINITY], and [f64::NEG_INFINITY] values are ignored.
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() || value.is_infinite() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + self.buffer.len() as u64
+ }
+
+ /// Merge the given TDigest into this one
+ pub fn merge(&mut self, other: &TDigestMut) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid {
+ mean: v,
+ weight: DEFAULT_WEIGHT,
+ });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid {
+ mean: v,
+ weight: DEFAULT_WEIGHT,
+ });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Freezes this TDigest into an immutable one.
+ pub fn freeze(mut self) -> TDigest {
+ self.compress();
+ TDigest {
+ k: self.k,
+ reverse_merge: self.reverse_merge,
+ min: self.min,
+ max: self.max,
+ centroids: self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ fn view(&mut self) -> TDigestView<'_> {
+ self.compress(); // side effect
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().quantile(rank)
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut total_size = 0;
+ if self.is_empty() || self.is_single_value() {
+ // 1 byte preamble
+ // + 1 byte serial version
+ // + 1 byte family
+ // + 2 bytes k
+ // + 1 byte flags
+ // + 2 bytes unused
+ total_size += size_of::<u64>();
+ } else {
+ // all of the above
+ // + 4 bytes num centroids
+ // + 4 bytes num buffered
+ total_size += size_of::<u64>() * 2;
+ }
+ if self.is_empty() {
+ // nothing more
+ } else if self.is_single_value() {
+ // + 8 bytes single value
+ total_size += size_of::<f64>();
+ } else {
+ // + 8 bytes min
+ // + 8 bytes max
+ total_size += size_of::<f64>() * 2;
+ // + (8+8) bytes per centroid
+ total_size += self.centroids.len() * (size_of::<f64>() + size_of::<u64>());
+ }
+
+ let mut bytes = Vec::with_capacity(total_size);
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ bytes.extend_from_slice(&self.k.to_le_bytes());
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ bytes.extend_from_slice(&0u16.to_le_bytes()); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ return bytes;
+ }
+ bytes.extend_from_slice(&(self.centroids.len() as u32).to_le_bytes());
+ bytes.extend_from_slice(&0u32.to_le_bytes()); // unused
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ bytes.extend_from_slice(&self.max.to_le_bytes());
+ for centroid in &self.centroids {
+ bytes.extend_from_slice(¢roid.mean.to_le_bytes());
+ bytes.extend_from_slice(¢roid.weight.get().to_le_bytes());
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// Supports reading format of the reference implementation (auto-detected) [^2].
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ /// [^2]: <https://github.com/tdunning/t-digest>
+ pub fn deserialize(bytes: &[u8], is_f32: bool) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(tag.to_string())
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ if preamble_longs == 0 && serial_version == 0 && family_id == 0 {
+ return Self::deserialize_compat(bytes);
+ }
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ if k < 10 {
+ return Err(SerdeError::InvalidParameter(format!(
+ "k must be at least 10, got {k}"
+ )));
+ }
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigestMut::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ check_non_nan(value, "single_value")?;
+ check_non_infinite(value, "single_value")?;
+ return Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: DEFAULT_WEIGHT,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ check_non_nan(min, "min")?;
+ check_non_nan(max, "max")?;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0u64;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ check_non_nan(mean, "centroid mean")?;
+ check_non_infinite(mean, "centroid")?;
+ let weight = check_nonzero(weight, "centroid weight")?;
+ centroids_weight += weight.get();
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ };
+ check_non_nan(value, "buffered_value mean")?;
+ check_non_infinite(value, "buffered_value mean")?;
+ buffer.push(value);
+ }
+ Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ // compatibility with the format of the reference implementation
+ // default byte order of ByteBuffer is used there, which is big endian
+ fn deserialize_compat(bytes: &[u8]) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat format"))
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let ty = cursor.read_u32::<BE>().map_err(make_error("type"))?;
+ match ty {
+ COMPAT_DOUBLE => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat double format"))
+ }
+ // compatibility with asBytes()
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ check_non_nan(min, "min in compat double format")?;
+ check_non_nan(max, "max in compat double format")?;
+ let k = cursor.read_f64::<BE>().map_err(make_error("k"))? as u16;
+ if k < 10 {
+ return Err(SerdeError::InvalidParameter(format!(
+ "k must be at least 10, got {k} in compat double format"
+ )));
+ }
+ let num_centroids = cursor
+ .read_u32::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0u64;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f64::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f64::<BE>().map_err(make_error("mean"))?;
+ let weight = check_nonzero(weight, "centroid weight in compat double format")?;
+ check_non_nan(mean, "centroid mean in compat double format")?;
+ check_non_infinite(mean, "centroid mean in compat double format")?;
+ total_weight += weight.get();
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ COMPAT_FLOAT => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat float format"))
+ }
+ // COMPAT_FLOAT: compatibility with asSmallBytes()
+ // reference implementation uses doubles for min and max
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ check_non_nan(min, "min in compat float format")?;
+ check_non_nan(max, "max in compat float format")?;
+ let k = cursor.read_f32::<BE>().map_err(make_error("k"))? as u16;
+ if k < 10 {
+ return Err(SerdeError::InvalidParameter(format!(
+ "k must be at least 10, got {k} in compat float format"
+ )));
+ }
+ // reference implementation stores capacities of the array of centroids and the
+ // buffer as shorts they can be derived from k in the constructor
+ cursor.read_u32::<BE>().map_err(make_error("<unused>"))?;
+ let num_centroids = cursor
+ .read_u16::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0u64;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f32::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f32::<BE>().map_err(make_error("mean"))? as f64;
+ let weight = check_nonzero(weight, "centroid weight in compat float format")?;
+ check_non_nan(mean, "centroid mean in compat float format")?;
+ check_non_infinite(mean, "centroid mean in compat float format")?;
+ total_weight += weight.get();
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ ty => Err(SerdeError::InvalidParameter(format!(
+ "unknown TDigest compat type {ty}",
+ ))),
+ }
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid {
+ mean: v,
+ weight: DEFAULT_WEIGHT,
+ });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = self.centroids[num_centroids - 1].weight() + c.weight();
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight();
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+/// Immutable (frozen) T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](super) for more details.
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+}
+
+impl TDigest {
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight
+ }
+
+ fn view(&self) -> TDigestView<'_> {
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+ self.view().quantile(rank)
+ }
+
+ /// Converts this immutable TDigest into a mutable one.
+ pub fn unfreeze(self) -> TDigestMut {
+ TDigestMut::make(
+ self.k,
+ self.reverse_merge,
+ self.min,
+ self.max,
+ self.centroids,
+ self.centroids_weight,
+ vec![],
+ )
+ }
+}
+
+struct TDigestView<'a> {
+ min: f64,
+ max: f64,
+ centroids: &'a [Centroid],
+ centroids_weight: u64,
+}
+
+impl TDigestView<'_> {
+ fn pmf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ let mut buckets = self.cdf(split_points)?;
+ for i in (1..buckets.len()).rev() {
+ buckets[i] -= buckets[i - 1];
+ }
+ Some(buckets)
+ }
+
+ fn cdf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.centroids.is_empty() {
+ return None;
+ }
+
+ let mut ranks = Vec::with_capacity(split_points.len() + 1);
+ for &p in split_points {
+ match self.rank(p) {
+ Some(rank) => ranks.push(rank),
+ None => unreachable!("checked non-empty above"),
+ }
+ }
+ ranks.push(1.0);
+ Some(ranks)
+ }
+
+ fn rank(&self, value: f64) -> Option<f64> {
+ debug_assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.centroids.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() == 1 {
+ return Some(0.5);
+ }
+
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight() / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight() / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight();
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight() / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight();
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight() / 2.;
+ weight_delta += self.centroids[upper].weight() / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ fn quantile(&self, rank: f64) -> Option<f64> {
+ debug_assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.centroids.is_empty() {
+ return None;
+ }
+
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight();
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight();
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight() + self.centroids[i + 1].weight()) / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight.get() == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight.get() == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight - (centroids_weight) - ((self.centroids[num_centroids - 1].weight()) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight() / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+}
+
+/// Checks the sequential validity of the given array of double values.
+/// They must be unique, monotonically increasing and not NaN.
+#[track_caller]
+fn check_split_points(split_points: &[f64]) {
+ let len = split_points.len();
+ if len == 1 && split_points[0].is_nan() {
+ panic!("split_points must not contain NaN values: {split_points:?}");
+ }
+ for i in 0..len - 1 {
+ if split_points[i] < split_points[i + 1] {
+ // we must use this positive condition because NaN comparisons are always false
+ continue;
+ }
+ panic!("split_points must be unique and monotonically increasing: {split_points:?}");
+ }
+}
Review Comment:
While C++ defines `check_split_points` as a private method, Java provides it as a public util: https://apache.github.io/datasketches-java/9.0.0/org/apache/datasketches/quantilescommon/QuantilesUtil.html#checkDoublesSplitPointsOrder(double%5B%5D)
Given that we require split_points to be unique, monotonically increasing and not NaN in `cdf`/`pmf` contract, I'd prefer to expose a fallible version of this method for users to check by themselves easily.
The remaining issue is where we should put these utilities. And this can be a follow-up anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622754228
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
This is reachable:
```rust
#[test]
fn test_two_value() {
let mut td1 = TDigest::new(10);
for _ in 0..10000 {
td1.update(f64::INFINITY);
}
println!("{:?}", td1.get_quantile(0.5));
}
```
Then we may need some total order for float anyway. Let me check how Java/C++ does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625321627
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Seems inevitable.
```rust
let mut td = TDigestMut::new(10);
for i in 0..10000 {
if i % 2 == 0 {
td.update(f64::INFINITY);
} else {
td.update(f64::NEG_INFINITY);
}
}
assert_eq!(td.quantile(0.5), Some(f64::NEG_INFINITY));
```
This would try to merge `-inf` and `inf` which is defined as NaN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625332149
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Fixes as many as possible in bebd87c & 243dc28.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "AlexanderSaydakov (via GitHub)" <gi...@apache.org>.
AlexanderSaydakov commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3666821042
> any reason we must use an integer for weight?
the total weight is the number of input items. weight of a centroid is the number of items this centroid represents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622632504
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean < value {
+ std::cmp::Ordering::Less
+ } else {
+ std::cmp::Ordering::Greater
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean > value {
+ std::cmp::Ordering::Greater
+ } else {
+ std::cmp::Ordering::Less
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ pub(super) fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ pub(super) fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> std::cmp::Ordering {
Review Comment:
Not quite. f64 does not have a total order in Rust, so I think we'd better define the cmp function explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2623711704
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
Review Comment:
Our use case has a major difference - Write to Vec should never fail.
So we can try to sync the use case and later build our own serde workflow, as discussed at https://github.com/apache/datasketches-rust/pull/24#issuecomment-3659560276.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3661202820
> > get_rank and get_quantile now takes &mut self because they call compress internally. I think we can add a "freezed" tdigest struct that will never updated then and already compressed, so that the freezed struct can be shared anywhere to test ranks and quantities without mut.
>
> I like this approach.
Added `TDigest` and `TDigestMut` for this purpose. See 7175d98 for more details. Also an internal `TDigestView<'a>` for sharing core logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3663996340
WRT +/-inf. This is truely bizarre. The KLL, Classic, REQ sketches reject NaNs (except in one very special case) As for +-Inf, I don't recall that they ever subtract points so the (inf - inf) should never happen. T-Digest is another animal entirely, and I can see where the subtraction of centroids would create a problem. i would reject all +/-Inf on the input, they really mess up data analysis IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2629016154
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Finally resolved by filtering out all inf/-inf/nan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3667628024
> > i would reject all +/-Inf on the input, they really mess up data analysis IMHO.
All filter out in the final commit.
@leerho @freakyzoidberg This PR should be considered finished now. Welcome to drop a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622681641
##########
src/tdigest/mod.rs:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! T-Digest implementation for estimating quantiles and ranks.
+//!
+//! The implementation in this library is based on the MergingDigest described in
+//! [Computing Extremely Accurate Quantiles Using t-Digests][paper] by Ted Dunning and Otmar Ertl.
+//!
+//! The implementation in this library has a few differences from the reference implementation
+//! associated with that paper:
+//!
+//! * Merge does not modify the input
+//! * Deserialization similar to other sketches in this library, although reading the reference
+//! implementation format is supported
+//!
+//! Unlike all other algorithms in the library, t-digest is empirical and has no mathematical
+//! basis for estimating its error and its results are dependent on the input data. However,
+//! for many common data distributions, it can produce excellent results. t-digest also operates
+//! only on numeric data and, unlike the quantiles family algorithms in the library which return
+//! quantile approximations from the input domain, t-digest interpolates values and will hold and
+//! return data points not seen in the input.
+//!
+//! The closest alternative to t-digest in this library is REQ sketch. It prioritizes one chosen
+//! side of the rank domain: either low rank accuracy or high rank accuracy. t-digest (in this
+//! implementation) prioritizes both ends of the rank domain and has lower accuracy towards the
+//! middle of the rank domain (median).
+//!
+//! Measurements show that t-digest is slightly biased (tends to underestimate low ranks and
+//! overestimate high ranks), while still doing very well close to the extremes. The effect seems
+//! to be more pronounced with more input values.
+//!
+//! For more information on the performance characteristics, see the
+//! [Datasketches page on t-digest](https://datasketches.apache.org/docs/tdigest/tdigest.html).
+//!
+//! [paper]: https://arxiv.org/abs/1902.04023
+
+mod serialization;
+mod sketch;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
Review Comment:
Updated in d90491d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622472594
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
Review Comment:
Nice crate, I think we can backport this to HLL as well in another PR.
##########
src/tdigest/mod.rs:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! T-Digest implementation for estimating quantiles and ranks.
+//!
+//! The implementation in this library is based on the MergingDigest described in
+//! [Computing Extremely Accurate Quantiles Using t-Digests][paper] by Ted Dunning and Otmar Ertl.
+//!
+//! The implementation in this library has a few differences from the reference implementation
+//! associated with that paper:
+//!
+//! * Merge does not modify the input
+//! * Deserialization similar to other sketches in this library, although reading the reference
+//! implementation format is supported
+//!
+//! Unlike all other algorithms in the library, t-digest is empirical and has no mathematical
+//! basis for estimating its error and its results are dependent on the input data. However,
+//! for many common data distributions, it can produce excellent results. t-digest also operates
+//! only on numeric data and, unlike the quantiles family algorithms in the library which return
+//! quantile approximations from the input domain, t-digest interpolates values and will hold and
+//! return data points not seen in the input.
+//!
+//! The closest alternative to t-digest in this library is REQ sketch. It prioritizes one chosen
+//! side of the rank domain: either low rank accuracy or high rank accuracy. t-digest (in this
+//! implementation) prioritizes both ends of the rank domain and has lower accuracy towards the
+//! middle of the rank domain (median).
+//!
+//! Measurements show that t-digest is slightly biased (tends to underestimate low ranks and
+//! overestimate high ranks), while still doing very well close to the extremes. The effect seems
+//! to be more pronounced with more input values.
+//!
+//! For more information on the performance characteristics, see the
+//! [Datasketches page on t-digest](https://datasketches.apache.org/docs/tdigest/tdigest.html).
+//!
+//! [paper]: https://arxiv.org/abs/1902.04023
+
+mod serialization;
+mod sketch;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
Review Comment:
I'm not a fan of defining a struct in a file and implementing it another.
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean < value {
+ std::cmp::Ordering::Less
+ } else {
+ std::cmp::Ordering::Greater
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean > value {
+ std::cmp::Ordering::Greater
+ } else {
+ std::cmp::Ordering::Less
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ pub(super) fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
Review Comment:
Can we assert somewhere that this won't grow over the capacity and re-alloc?
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
Review Comment:
Can we assert somewhere that this won't grow over the capacity and re-alloc?
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,453 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+use crate::tdigest::{Centroid, TDigest};
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ pub(super) fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean < value {
+ std::cmp::Ordering::Less
+ } else {
+ std::cmp::Ordering::Greater
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| {
+ if c.mean > value {
+ std::cmp::Ordering::Greater
+ } else {
+ std::cmp::Ordering::Less
+ }
+ })
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ pub(super) fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ pub(super) fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> std::cmp::Ordering {
Review Comment:
Can we implement this via Ord / PartialOrd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622905327
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
Review Comment:
Sorry, I haven't read the paper yet. Would it be possible to add a brief explanation to this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3663119316
> > cc @notfilippo @freakyzoidberg
> > cc @leerho based on [#2 (comment)](https://github.com/apache/datasketches-rust/issues/2#issuecomment-3657395474), I made this impl as a combination of C++ & Java version. Welcome to give it a review. I'll add serde support today or tomorrow. But I'm still not quite sure what CDF and PMF are. It's possible to convey C++'s impl as is but I wonder a real world use case to understand its definition and usage.
>
> Without the x-serde this is quite hard to confirm if the synopsis are compatible But that can be done later a the implem progress
>
> Note I am no mathematician - but my humble understanding
>
> ### PMF -> Probability Mass Function
> It returns the approximate fraction of data points (mass) that fall into specific "bins" or intervals. You give it an array of split points and it returns an array of mass fraction that sum up to 1.
>
> it's a histogram generator of sort
>
> ### CDF -> Cumulative Distribution Function
> It returns the approximate fraction of data points that are less than (or equal to) each split point. CDF ~= running running sum of the PMF
Thanks for your explaination @freakyzoidberg! Generally got it now.
This PR is now ready for merge. Some follow-up improvement are documented above. Currently, the functionality is aligned with C++/Java impls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625274947
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,1097 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{BE, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+/// The default value of K if one is not specified.
+const DEFAULT_K: u16 = 200;
+/// Multiplier for buffer size relative to centroids capacity.
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone)]
+pub struct TDigestMut {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigestMut {
+ fn default() -> Self {
+ TDigestMut::new(DEFAULT_K)
+ }
+}
+
+impl TDigestMut {
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigestMut {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given TDigest into this one
+ pub fn merge(&mut self, other: &TDigestMut) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Freezes this TDigest into an immutable one.
+ pub fn freeze(mut self) -> TDigest {
+ self.compress();
+ TDigest {
+ k: self.k,
+ reverse_merge: self.reverse_merge,
+ min: self.min,
+ max: self.max,
+ centroids: self.centroids,
+ centroids_weight: self.centroids_weight,
+ centroids_capacity: self.centroids_capacity,
+ }
+ }
+
+ fn view(&mut self) -> TDigestView<'_> {
+ self.compress(); // side effect
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().quantile(rank)
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut total_size = 0;
+ if self.is_empty() || self.is_single_value() {
+ // 1 byte preamble
+ // + 1 byte serial version
+ // + 1 byte family
+ // + 2 bytes k
+ // + 1 byte flags
+ // + 2 bytes unused
+ total_size += size_of::<u64>();
+ } else {
+ // all of the above
+ // + 4 bytes num centroids
+ // + 4 bytes num buffered
+ total_size += size_of::<u64>() * 2;
+ }
+ if self.is_empty() {
+ // nothing more
+ } else if self.is_single_value() {
+ // + 8 bytes single value
+ total_size += size_of::<f64>();
+ } else {
+ // + 8 bytes min
+ // + 8 bytes max
+ total_size += size_of::<f64>() * 2;
+ // + (8+8) bytes per centroid
+ total_size += self.centroids.len() * (size_of::<f64>() + size_of::<u64>());
+ }
+
+ let mut bytes = Vec::with_capacity(total_size);
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ bytes.extend_from_slice(&self.k.to_le_bytes());
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ bytes.extend_from_slice(&0u16.to_le_bytes()); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ return bytes;
+ }
+ bytes.extend_from_slice(&(self.centroids.len() as u32).to_le_bytes());
+ bytes.extend_from_slice(&0u32.to_le_bytes()); // unused
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ bytes.extend_from_slice(&self.max.to_le_bytes());
+ for centroid in &self.centroids {
+ bytes.extend_from_slice(¢roid.mean.to_le_bytes());
+ bytes.extend_from_slice(¢roid.weight.to_le_bytes());
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// Supports reading format of the reference implementation (auto-detected) [^2].
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ /// [^2]: https://github.com/tdunning/t-digest
+ pub fn deserialize(bytes: &[u8], is_f32: bool) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(tag.to_string())
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ if preamble_longs == 0 && serial_version == 0 && family_id == 0 {
+ return Self::deserialize_compat(bytes);
+ }
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigestMut::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ // compatibility with the format of the reference implementation
+ // default byte order of ByteBuffer is used there, which is big endian
+ fn deserialize_compat(bytes: &[u8]) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat format"))
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let ty = cursor.read_u32::<BE>().map_err(make_error("type"))?;
+ match ty {
+ COMPAT_DOUBLE => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat double format"))
+ }
+ // compatibility with asBytes()
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f64::<BE>().map_err(make_error("k"))? as u16;
+ let num_centroids = cursor
+ .read_u32::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f64::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f64::<BE>().map_err(make_error("mean"))?;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ COMPAT_FLOAT => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat float format"))
+ }
+ // COMPAT_FLOAT: compatibility with asSmallBytes()
+ // reference implementation uses doubles for min and max
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f32::<BE>().map_err(make_error("k"))? as u16;
+ // reference implementation stores capacities of the array of centroids and the
+ // buffer as shorts they can be derived from k in the constructor
+ cursor.read_u32::<BE>().map_err(make_error("<unused>"))?;
+ let num_centroids = cursor
+ .read_u16::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f32::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f32::<BE>().map_err(make_error("mean"))? as f64;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ ty => Err(SerdeError::InvalidParameter(format!(
+ "unknown TDigest compat type {ty}",
+ ))),
+ }
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+/// Immutable (frozen) T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+}
+
+impl TDigest {
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight
+ }
+
+ fn view(&self) -> TDigestView<'_> {
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+ self.view().quantile(rank)
+ }
+
+ /// Converts this immutable TDigest into a mutable one.
+ pub fn into_mut(mut self) -> TDigestMut {
Review Comment:
I'm not very satisfied with this name. But `unfreeze` sounds weird. Open to any good alternative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622410785
##########
src/tdigest/serialization.rs:
##########
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::{Centroid, TDigest};
+
+const PREAMBLE_LONGS_EMPTY_OR_SINGLE: u8 = 1;
+const PREAMBLE_LONGS_MULTIPLE: u8 = 2;
+const SERIAL_VERSION: u8 = 1;
+const TDIGEST_FAMILY_ID: u8 = 20;
+const FLAGS_IS_EMPTY: u8 = 1 << 0;
+const FLAGS_IS_SINGLE_VALUE: u8 = 1 << 1;
+const FLAGS_REVERSE_MERGE: u8 = 1 << 2;
+
+impl TDigest {
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
Review Comment:
Should calculate the size and reverse the space. Otherwise this call would panic. Also pre-reserve can avoid realloc on the fly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622938899
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
Review Comment:
I don't know either. This is from Java/C++ impl and I'm afraid add this fudge rather than use `k * 2` may exceed the length assumption https://github.com/apache/datasketches-rust/pull/23#discussion_r2622259722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622970627
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,673 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{ByteOrder, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: u16 = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigest {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given t-Digest into this one
+ pub fn merge(&mut self, other: &TDigest) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn get_rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.compress(); // side effect
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+
+ // left tail
+ let first_mean = self.centroids[0].mean;
+ if value < first_mean {
+ if first_mean - self.min > 0. {
+ return Some(if value == self.min {
+ 0.5 / centroids_weight
+ } else {
+ 1. + (((value - self.min) / (first_mean - self.min))
+ * ((self.centroids[0].weight as f64 / 2.) - 1.))
+ });
+ }
+ return Some(0.); // should never happen
+ }
+
+ // right tail
+ let last_mean = self.centroids[num_centroids - 1].mean;
+ if value > last_mean {
+ if self.max - last_mean > 0. {
+ return Some(if value == self.max {
+ 1. - (0.5 / centroids_weight)
+ } else {
+ 1.0 - ((1.0
+ + (((self.max - value) / (self.max - last_mean))
+ * ((self.centroids[num_centroids - 1].weight as f64 / 2.) - 1.)))
+ / centroids_weight)
+ });
+ }
+ return Some(1.); // should never happen
+ }
+
+ let mut lower = self
+ .centroids
+ .binary_search_by(|c| centroid_lower_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(lower, num_centroids, "get_rank: lower == end");
+ let mut upper = self
+ .centroids
+ .binary_search_by(|c| centroid_upper_bound(c, value))
+ .unwrap_or_else(identity);
+ debug_assert_ne!(upper, 0, "get_rank: upper == begin");
+ if value < self.centroids[lower].mean {
+ lower -= 1;
+ }
+ if (upper == num_centroids) || (self.centroids[upper - 1].mean >= value) {
+ upper -= 1;
+ }
+
+ let mut weight_below = 0.;
+ let mut i = 0;
+ while i < lower {
+ weight_below += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_below += self.centroids[lower].weight as f64 / 2.;
+
+ let mut weight_delta = 0.;
+ while i < upper {
+ weight_delta += self.centroids[i].weight as f64;
+ i += 1;
+ }
+ weight_delta -= self.centroids[lower].weight as f64 / 2.;
+ weight_delta += self.centroids[upper].weight as f64 / 2.;
+ Some(
+ if self.centroids[upper].mean - self.centroids[lower].mean > 0. {
+ (weight_below
+ + (weight_delta * (value - self.centroids[lower].mean)
+ / (self.centroids[upper].mean - self.centroids[lower].mean)))
+ / centroids_weight
+ } else {
+ (weight_below + weight_delta / 2.) / centroids_weight
+ },
+ )
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn get_quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.compress(); // side effect
+ if self.centroids.len() == 1 {
+ return Some(self.centroids[0].mean);
+ }
+
+ // at least 2 centroids
+ let centroids_weight = self.centroids_weight as f64;
+ let num_centroids = self.centroids.len();
+ let weight = rank * centroids_weight;
+ if weight < 1. {
+ return Some(self.min);
+ }
+ if weight > centroids_weight - 1. {
+ return Some(self.max);
+ }
+ let first_weight = self.centroids[0].weight as f64;
+ if first_weight > 1. && weight < first_weight / 2. {
+ return Some(
+ self.min
+ + (((weight - 1.) / ((first_weight / 2.) - 1.))
+ * (self.centroids[0].mean - self.min)),
+ );
+ }
+ let last_weight = self.centroids[num_centroids - 1].weight as f64;
+ if last_weight > 1. && (centroids_weight - weight <= last_weight / 2.) {
+ return Some(
+ self.max
+ + (((centroids_weight - weight - 1.) / ((last_weight / 2.) - 1.))
+ * (self.max - self.centroids[num_centroids - 1].mean)),
+ );
+ }
+
+ // interpolate between extremes
+ let mut weight_so_far = first_weight / 2.;
+ for i in 0..(num_centroids - 1) {
+ let dw = (self.centroids[i].weight + self.centroids[i + 1].weight) as f64 / 2.;
+ if weight_so_far + dw > weight {
+ // the target weight is between centroids i and i+1
+ let mut left_weight = 0.;
+ if self.centroids[i].weight == 1 {
+ if weight - weight_so_far < 0.5 {
+ return Some(self.centroids[i].mean);
+ }
+ left_weight = 0.5;
+ }
+ let mut right_weight = 0.;
+ if self.centroids[i + 1].weight == 1 {
+ if weight_so_far + dw - weight < 0.5 {
+ return Some(self.centroids[i + 1].mean);
+ }
+ right_weight = 0.5;
+ }
+ let w1 = weight - weight_so_far - left_weight;
+ let w2 = weight_so_far + dw - weight - right_weight;
+ return Some(weighted_average(
+ self.centroids[i].mean,
+ w1,
+ self.centroids[i + 1].mean,
+ w2,
+ ));
+ }
+ weight_so_far += dw;
+ }
+
+ let w1 = weight
+ - (self.centroids_weight as f64)
+ - ((self.centroids[num_centroids - 1].weight as f64) / 2.);
+ let w2 = (self.centroids[num_centroids - 1].weight as f64 / 2.) - w1;
+ Some(weighted_average(
+ self.centroids[num_centroids - 1].mean,
+ w1,
+ self.max,
+ w2,
+ ))
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut bytes = vec![];
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
+ LE::write_u32(&mut bytes, 0); // unused
+ LE::write_f64(&mut bytes, self.min);
+ LE::write_f64(&mut bytes, self.max);
+ for centroid in &self.centroids {
+ LE::write_f64(&mut bytes, centroid.mean);
+ LE::write_u64(&mut bytes, centroid.weight);
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ pub fn deserialize(bytes: &[u8], is_float: bool) -> Result<Self, SerdeError> {
+ let make_error = |tag: &'static str| move |_| SerdeError::InsufficientData(tag.to_string());
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ // TODO: Support reading format of the reference implementation
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigest::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigest::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_float {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_float {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigest::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+fn centroid_cmp(a: &Centroid, b: &Centroid) -> Ordering {
+ match a.mean.partial_cmp(&b.mean) {
+ Some(order) => order,
+ None => unreachable!("NaN values should never be present in centroids"),
Review Comment:
Perhaps a reasonable solution is to just use Inf when merging Inf and Inf. This should be the only case that can produce NaN, and we're sorted already, so the merge must happen between two adjacent `inf`s. And we can use `inf` in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3659797022
> Is there any way we can have a TDigest<T: num::Float>? I think in some cases having less precision for half of the memory is desirable.
Possible. The tricky part is around overflow/underflow during computation and serde.
And the more tricky part is that it's not only about the value, but also the weight (u64 for f64, u32 for f32). But I'm considering why not just use f64/f32 for weight - we're now often casting weight to f64 and while representing u64 as f64 is lossless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3673216132
See https://github.com/apache/datasketches-rust/tree/4f5f266/tests/serialization_test_data and @notfilippo may prefer check in the snapshot files as in (https://github.com/apache/datasketches-rust/pull/21#issuecomment-3656421311) and what datasketches-go does now (https://github.com/apache/datasketches-go/tree/f7bc4b1d/serialization_test_data).
But I'd prefer Java/C++'s approach so let me prepare a patch for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625277905
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,1097 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{BE, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+/// The default value of K if one is not specified.
+const DEFAULT_K: u16 = 200;
+/// Multiplier for buffer size relative to centroids capacity.
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone)]
+pub struct TDigestMut {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigestMut {
+ fn default() -> Self {
+ TDigestMut::new(DEFAULT_K)
+ }
+}
+
+impl TDigestMut {
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigestMut {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given TDigest into this one
+ pub fn merge(&mut self, other: &TDigestMut) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Freezes this TDigest into an immutable one.
+ pub fn freeze(mut self) -> TDigest {
+ self.compress();
+ TDigest {
+ k: self.k,
+ reverse_merge: self.reverse_merge,
+ min: self.min,
+ max: self.max,
+ centroids: self.centroids,
+ centroids_weight: self.centroids_weight,
+ centroids_capacity: self.centroids_capacity,
+ }
+ }
+
+ fn view(&mut self) -> TDigestView<'_> {
+ self.compress(); // side effect
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().quantile(rank)
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut total_size = 0;
+ if self.is_empty() || self.is_single_value() {
+ // 1 byte preamble
+ // + 1 byte serial version
+ // + 1 byte family
+ // + 2 bytes k
+ // + 1 byte flags
+ // + 2 bytes unused
+ total_size += size_of::<u64>();
+ } else {
+ // all of the above
+ // + 4 bytes num centroids
+ // + 4 bytes num buffered
+ total_size += size_of::<u64>() * 2;
+ }
+ if self.is_empty() {
+ // nothing more
+ } else if self.is_single_value() {
+ // + 8 bytes single value
+ total_size += size_of::<f64>();
+ } else {
+ // + 8 bytes min
+ // + 8 bytes max
+ total_size += size_of::<f64>() * 2;
+ // + (8+8) bytes per centroid
+ total_size += self.centroids.len() * (size_of::<f64>() + size_of::<u64>());
+ }
+
+ let mut bytes = Vec::with_capacity(total_size);
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ bytes.extend_from_slice(&self.k.to_le_bytes());
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ bytes.extend_from_slice(&0u16.to_le_bytes()); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ return bytes;
+ }
+ bytes.extend_from_slice(&(self.centroids.len() as u32).to_le_bytes());
+ bytes.extend_from_slice(&0u32.to_le_bytes()); // unused
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ bytes.extend_from_slice(&self.max.to_le_bytes());
+ for centroid in &self.centroids {
+ bytes.extend_from_slice(¢roid.mean.to_le_bytes());
+ bytes.extend_from_slice(¢roid.weight.to_le_bytes());
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// Supports reading format of the reference implementation (auto-detected) [^2].
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ /// [^2]: https://github.com/tdunning/t-digest
+ pub fn deserialize(bytes: &[u8], is_f32: bool) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(tag.to_string())
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ if preamble_longs == 0 && serial_version == 0 && family_id == 0 {
+ return Self::deserialize_compat(bytes);
+ }
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigestMut::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ // compatibility with the format of the reference implementation
+ // default byte order of ByteBuffer is used there, which is big endian
+ fn deserialize_compat(bytes: &[u8]) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat format"))
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let ty = cursor.read_u32::<BE>().map_err(make_error("type"))?;
+ match ty {
+ COMPAT_DOUBLE => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat double format"))
+ }
+ // compatibility with asBytes()
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f64::<BE>().map_err(make_error("k"))? as u16;
+ let num_centroids = cursor
+ .read_u32::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f64::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f64::<BE>().map_err(make_error("mean"))?;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ COMPAT_FLOAT => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat float format"))
+ }
+ // COMPAT_FLOAT: compatibility with asSmallBytes()
+ // reference implementation uses doubles for min and max
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f32::<BE>().map_err(make_error("k"))? as u16;
+ // reference implementation stores capacities of the array of centroids and the
+ // buffer as shorts they can be derived from k in the constructor
+ cursor.read_u32::<BE>().map_err(make_error("<unused>"))?;
+ let num_centroids = cursor
+ .read_u16::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f32::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f32::<BE>().map_err(make_error("mean"))? as f64;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ ty => Err(SerdeError::InvalidParameter(format!(
+ "unknown TDigest compat type {ty}",
+ ))),
+ }
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+/// Immutable (frozen) T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+}
+
+impl TDigest {
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight
+ }
+
+ fn view(&self) -> TDigestView<'_> {
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+ self.view().quantile(rank)
+ }
+
+ /// Converts this immutable TDigest into a mutable one.
+ pub fn into_mut(mut self) -> TDigestMut {
Review Comment:
To be symmetric with `TDigestMut::freeze`. I changed it to `unfreeze` now. But I find it somewhat rare :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2625277905
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,1097 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use byteorder::{BE, LE, ReadBytesExt};
+use std::cmp::Ordering;
+use std::convert::identity;
+use std::io::Cursor;
+
+use crate::error::SerdeError;
+use crate::tdigest::serialization::*;
+
+/// The default value of K if one is not specified.
+const DEFAULT_K: u16 = 200;
+/// Multiplier for buffer size relative to centroids capacity.
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+#[derive(Debug, Clone)]
+pub struct TDigestMut {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigestMut {
+ fn default() -> Self {
+ TDigestMut::new(DEFAULT_K)
+ }
+}
+
+impl TDigestMut {
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: u16) -> Self {
+ Self::make(
+ k,
+ false,
+ f64::INFINITY,
+ f64::NEG_INFINITY,
+ vec![],
+ 0,
+ vec![],
+ )
+ }
+
+ // for deserialization
+ fn make(
+ k: u16,
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+ mut centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ mut buffer: Vec<f64>,
+ ) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k as usize * 2) + fudge;
+
+ centroids.reserve(centroids_capacity);
+ buffer.reserve(centroids_capacity * BUFFER_MULTIPLIER);
+
+ TDigestMut {
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ centroids_capacity,
+ buffer,
+ }
+ }
+
+ /// Update this TDigest with the given value (`NaN` values are ignored).
+ pub fn update(&mut self, value: f64) {
+ if value.is_nan() {
+ return;
+ }
+
+ if self.buffer.len() == self.centroids_capacity * BUFFER_MULTIPLIER {
+ self.compress();
+ }
+
+ self.buffer.push(value);
+ self.min = self.min.min(value);
+ self.max = self.max.max(value);
+ }
+
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty() && self.buffer.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight + (self.buffer.len() as u64)
+ }
+
+ /// Merge the given TDigest into this one
+ pub fn merge(&mut self, other: &TDigestMut) {
+ if other.is_empty() {
+ return;
+ }
+
+ let mut tmp = Vec::with_capacity(
+ self.centroids.len() + self.buffer.len() + other.centroids.len() + other.buffer.len(),
+ );
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &v in &other.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ for &c in &other.centroids {
+ tmp.push(c);
+ }
+ self.do_merge(tmp, self.buffer.len() as u64 + other.total_weight())
+ }
+
+ /// Freezes this TDigest into an immutable one.
+ pub fn freeze(mut self) -> TDigest {
+ self.compress();
+ TDigest {
+ k: self.k,
+ reverse_merge: self.reverse_merge,
+ min: self.min,
+ max: self.max,
+ centroids: self.centroids,
+ centroids_weight: self.centroids_weight,
+ centroids_capacity: self.centroids_capacity,
+ }
+ }
+
+ fn view(&mut self) -> TDigestView<'_> {
+ self.compress(); // side effect
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&mut self, split_points: &[f64]) -> Option<Vec<f64>> {
+ check_split_points(split_points);
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&mut self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+
+ if self.is_empty() {
+ return None;
+ }
+ if value < self.min {
+ return Some(0.0);
+ }
+ if value > self.max {
+ return Some(1.0);
+ }
+ // one centroid and value == min == max
+ if self.centroids.len() + self.buffer.len() == 1 {
+ return Some(0.5);
+ }
+
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&mut self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+
+ if self.is_empty() {
+ return None;
+ }
+
+ self.view().quantile(rank)
+ }
+
+ /// Serializes this TDigest to bytes.
+ pub fn serialize(&mut self) -> Vec<u8> {
+ self.compress();
+
+ let mut total_size = 0;
+ if self.is_empty() || self.is_single_value() {
+ // 1 byte preamble
+ // + 1 byte serial version
+ // + 1 byte family
+ // + 2 bytes k
+ // + 1 byte flags
+ // + 2 bytes unused
+ total_size += size_of::<u64>();
+ } else {
+ // all of the above
+ // + 4 bytes num centroids
+ // + 4 bytes num buffered
+ total_size += size_of::<u64>() * 2;
+ }
+ if self.is_empty() {
+ // nothing more
+ } else if self.is_single_value() {
+ // + 8 bytes single value
+ total_size += size_of::<f64>();
+ } else {
+ // + 8 bytes min
+ // + 8 bytes max
+ total_size += size_of::<f64>() * 2;
+ // + (8+8) bytes per centroid
+ total_size += self.centroids.len() * (size_of::<f64>() + size_of::<u64>());
+ }
+
+ let mut bytes = Vec::with_capacity(total_size);
+ bytes.push(match self.total_weight() {
+ 0 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ 1 => PREAMBLE_LONGS_EMPTY_OR_SINGLE,
+ _ => PREAMBLE_LONGS_MULTIPLE,
+ });
+ bytes.push(SERIAL_VERSION);
+ bytes.push(TDIGEST_FAMILY_ID);
+ bytes.extend_from_slice(&self.k.to_le_bytes());
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ bytes.extend_from_slice(&0u16.to_le_bytes()); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ return bytes;
+ }
+ bytes.extend_from_slice(&(self.centroids.len() as u32).to_le_bytes());
+ bytes.extend_from_slice(&0u32.to_le_bytes()); // unused
+ bytes.extend_from_slice(&self.min.to_le_bytes());
+ bytes.extend_from_slice(&self.max.to_le_bytes());
+ for centroid in &self.centroids {
+ bytes.extend_from_slice(¢roid.mean.to_le_bytes());
+ bytes.extend_from_slice(¢roid.weight.to_le_bytes());
+ }
+ bytes
+ }
+
+ /// Deserializes a TDigest from bytes.
+ ///
+ /// Supports reading compact format with (float, int) centroids as opposed to (double, long) to
+ /// represent (mean, weight). [^1]
+ ///
+ /// Supports reading format of the reference implementation (auto-detected) [^2].
+ ///
+ /// [^1]: This is to support reading the `tdigest<float>` format from the C++ implementation.
+ /// [^2]: https://github.com/tdunning/t-digest
+ pub fn deserialize(bytes: &[u8], is_f32: bool) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(tag.to_string())
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let preamble_longs = cursor.read_u8().map_err(make_error("preamble_longs"))?;
+ let serial_version = cursor.read_u8().map_err(make_error("serial_version"))?;
+ let family_id = cursor.read_u8().map_err(make_error("family_id"))?;
+ if family_id != TDIGEST_FAMILY_ID {
+ if preamble_longs == 0 && serial_version == 0 && family_id == 0 {
+ return Self::deserialize_compat(bytes);
+ }
+ return Err(SerdeError::InvalidFamily(format!(
+ "expected {} (TDigest), got {}",
+ TDIGEST_FAMILY_ID, family_id
+ )));
+ }
+ if serial_version != SERIAL_VERSION {
+ return Err(SerdeError::UnsupportedVersion(format!(
+ "expected {}, got {}",
+ SERIAL_VERSION, serial_version
+ )));
+ }
+ let k = cursor.read_u16::<LE>().map_err(make_error("k"))?;
+ let flags = cursor.read_u8().map_err(make_error("flags"))?;
+ let is_empty = (flags & FLAGS_IS_EMPTY) != 0;
+ let is_single_value = (flags & FLAGS_IS_SINGLE_VALUE) != 0;
+ let expected_preamble_longs = if is_empty || is_single_value {
+ PREAMBLE_LONGS_EMPTY_OR_SINGLE
+ } else {
+ PREAMBLE_LONGS_MULTIPLE
+ };
+ if preamble_longs != expected_preamble_longs {
+ return Err(SerdeError::MalformedData(format!(
+ "expected preamble_longs to be {}, got {}",
+ expected_preamble_longs, preamble_longs
+ )));
+ }
+ cursor.read_u16::<LE>().map_err(make_error("<unused>"))?; // unused
+ if is_empty {
+ return Ok(TDigestMut::new(k));
+ }
+
+ let reverse_merge = (flags & FLAGS_REVERSE_MERGE) != 0;
+ if is_single_value {
+ let value = if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("single_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("single_value"))?
+ };
+ return Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ value,
+ value,
+ vec![Centroid {
+ mean: value,
+ weight: 1,
+ }],
+ 1,
+ vec![],
+ ));
+ }
+ let num_centroids = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_centroids"))? as usize;
+ let num_buffered = cursor
+ .read_u32::<LE>()
+ .map_err(make_error("num_buffered"))? as usize;
+ let (min, max) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("min"))? as f64,
+ cursor.read_f32::<LE>().map_err(make_error("max"))? as f64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("min"))?,
+ cursor.read_f64::<LE>().map_err(make_error("max"))?,
+ )
+ };
+ let mut centroids = Vec::with_capacity(num_centroids);
+ let mut centroids_weight = 0;
+ for _ in 0..num_centroids {
+ let (mean, weight) = if is_f32 {
+ (
+ cursor.read_f32::<LE>().map_err(make_error("mean"))? as f64,
+ cursor.read_u32::<LE>().map_err(make_error("weight"))? as u64,
+ )
+ } else {
+ (
+ cursor.read_f64::<LE>().map_err(make_error("mean"))?,
+ cursor.read_u64::<LE>().map_err(make_error("weight"))?,
+ )
+ };
+ centroids_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ let mut buffer = Vec::with_capacity(num_buffered);
+ for _ in 0..num_buffered {
+ buffer.push(if is_f32 {
+ cursor
+ .read_f32::<LE>()
+ .map_err(make_error("buffered_value"))? as f64
+ } else {
+ cursor
+ .read_f64::<LE>()
+ .map_err(make_error("buffered_value"))?
+ })
+ }
+ Ok(TDigestMut::make(
+ k,
+ reverse_merge,
+ min,
+ max,
+ centroids,
+ centroids_weight,
+ buffer,
+ ))
+ }
+
+ // compatibility with the format of the reference implementation
+ // default byte order of ByteBuffer is used there, which is big endian
+ fn deserialize_compat(bytes: &[u8]) -> Result<Self, SerdeError> {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat format"))
+ }
+
+ let mut cursor = Cursor::new(bytes);
+
+ let ty = cursor.read_u32::<BE>().map_err(make_error("type"))?;
+ match ty {
+ COMPAT_DOUBLE => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat double format"))
+ }
+ // compatibility with asBytes()
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f64::<BE>().map_err(make_error("k"))? as u16;
+ let num_centroids = cursor
+ .read_u32::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f64::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f64::<BE>().map_err(make_error("mean"))?;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ COMPAT_FLOAT => {
+ fn make_error(tag: &'static str) -> impl FnOnce(std::io::Error) -> SerdeError {
+ move |_| SerdeError::InsufficientData(format!("{tag} in compat float format"))
+ }
+ // COMPAT_FLOAT: compatibility with asSmallBytes()
+ // reference implementation uses doubles for min and max
+ let min = cursor.read_f64::<BE>().map_err(make_error("min"))?;
+ let max = cursor.read_f64::<BE>().map_err(make_error("max"))?;
+ let k = cursor.read_f32::<BE>().map_err(make_error("k"))? as u16;
+ // reference implementation stores capacities of the array of centroids and the
+ // buffer as shorts they can be derived from k in the constructor
+ cursor.read_u32::<BE>().map_err(make_error("<unused>"))?;
+ let num_centroids = cursor
+ .read_u16::<BE>()
+ .map_err(make_error("num_centroids"))?
+ as usize;
+ let mut total_weight = 0;
+ let mut centroids = Vec::with_capacity(num_centroids);
+ for _ in 0..num_centroids {
+ let weight = cursor.read_f32::<BE>().map_err(make_error("weight"))? as u64;
+ let mean = cursor.read_f32::<BE>().map_err(make_error("mean"))? as f64;
+ total_weight += weight;
+ centroids.push(Centroid { mean, weight });
+ }
+ Ok(TDigestMut::make(
+ k,
+ false,
+ min,
+ max,
+ centroids,
+ total_weight,
+ vec![],
+ ))
+ }
+ ty => Err(SerdeError::InvalidParameter(format!(
+ "unknown TDigest compat type {ty}",
+ ))),
+ }
+ }
+
+ fn is_single_value(&self) -> bool {
+ self.total_weight() == 1
+ }
+
+ /// Process buffered values and merge centroids if needed.
+ fn compress(&mut self) {
+ if self.buffer.is_empty() {
+ return;
+ }
+ let mut tmp = Vec::with_capacity(self.buffer.len() + self.centroids.len());
+ for &v in &self.buffer {
+ tmp.push(Centroid { mean: v, weight: 1 });
+ }
+ self.do_merge(tmp, self.buffer.len() as u64)
+ }
+
+ /// Merges the given buffer of centroids into this TDigest.
+ ///
+ /// # Contract
+ ///
+ /// * `buffer` must have at least one centroid.
+ /// * `buffer` is generated from `self.buffer`, and thus:
+ /// * No `NAN` values are present in `buffer`.
+ /// * We should clear `self.buffer` after merging.
+ fn do_merge(&mut self, mut buffer: Vec<Centroid>, weight: u64) {
+ buffer.extend(std::mem::take(&mut self.centroids));
+ buffer.sort_by(centroid_cmp);
+ if self.reverse_merge {
+ buffer.reverse();
+ }
+ self.centroids_weight += weight;
+
+ let mut num_centroids = 0;
+ let len = buffer.len();
+ self.centroids.push(buffer[0]);
+ num_centroids += 1;
+ let mut current = 1;
+ let mut weight_so_far = 0.;
+ while current < len {
+ let c = buffer[current];
+ let proposed_weight = (self.centroids[num_centroids - 1].weight + c.weight) as f64;
+ let mut add_this = false;
+ if (current != 1) && (current != (len - 1)) {
+ let centroids_weight = self.centroids_weight as f64;
+ let q0 = weight_so_far / centroids_weight;
+ let q2 = (weight_so_far + proposed_weight) / centroids_weight;
+ let normalizer = scale_function::normalizer((2 * self.k) as f64, centroids_weight);
+ add_this = proposed_weight
+ <= (centroids_weight
+ * scale_function::max(q0, normalizer)
+ .min(scale_function::max(q2, normalizer)));
+ }
+ if add_this {
+ // merge into existing centroid
+ self.centroids[num_centroids - 1].add(c);
+ } else {
+ // copy to a new centroid
+ weight_so_far += self.centroids[num_centroids - 1].weight as f64;
+ self.centroids.push(c);
+ num_centroids += 1;
+ }
+ current += 1;
+ }
+
+ if self.reverse_merge {
+ self.centroids.reverse();
+ }
+ self.min = self.min.min(self.centroids[0].mean);
+ self.max = self.max.max(self.centroids[num_centroids - 1].mean);
+ self.reverse_merge = !self.reverse_merge;
+ self.buffer.clear();
+ }
+}
+
+/// Immutable (frozen) T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](self) for more details.
+pub struct TDigest {
+ k: u16,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+}
+
+impl TDigest {
+ /// Returns parameter k (compression) that was used to configure this TDigest.
+ pub fn k(&self) -> u16 {
+ self.k
+ }
+
+ /// Returns true if TDigest has not seen any data.
+ pub fn is_empty(&self) -> bool {
+ self.centroids.is_empty()
+ }
+
+ /// Returns minimum value seen by TDigest; `None` if TDigest is empty.
+ pub fn min_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.min)
+ }
+ }
+
+ /// Returns maximum value seen by TDigest; `None` if TDigest is empty.
+ pub fn max_value(&self) -> Option<f64> {
+ if self.is_empty() {
+ None
+ } else {
+ Some(self.max)
+ }
+ }
+
+ /// Returns total weight.
+ pub fn total_weight(&self) -> u64 {
+ self.centroids_weight
+ }
+
+ fn view(&self) -> TDigestView<'_> {
+ TDigestView {
+ min: self.min,
+ max: self.max,
+ centroids: &self.centroids,
+ centroids_weight: self.centroids_weight,
+ }
+ }
+
+ /// Returns an approximation to the Cumulative Distribution Function (CDF), which is the
+ /// cumulative analog of the PMF, of the input stream given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals.
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles, which are a consecutive approximation to the CDF of the input
+ /// stream given the split points. The value at array position j of the returned CDF array
+ /// is the sum of the returned values in positions 0 through j of the returned PMF array.
+ /// This can be viewed as array of ranks of the given split points plus one more value that
+ /// is always 1.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn cdf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().cdf(split_points)
+ }
+
+ /// Returns an approximation to the Probability Mass Function (PMF) of the input stream
+ /// given a set of split points.
+ ///
+ /// # Arguments
+ ///
+ /// * `split_points`: An array of _m_ unique, monotonically increasing values that divide the
+ /// input domain into _m+1_ consecutive disjoint intervals (bins).
+ ///
+ /// # Returns
+ ///
+ /// An array of m+1 doubles each of which is an approximation to the fraction of the input
+ /// stream values (the mass) that fall into one of those intervals.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If `split_points` is not unique, not monotonically increasing, or contains `NaN` values.
+ pub fn pmf(&self, split_points: &[f64]) -> Option<Vec<f64>> {
+ self.view().pmf(split_points)
+ }
+
+ /// Compute approximate normalized rank (from 0 to 1 inclusive) of the given value.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If the value is `NaN`.
+ pub fn rank(&self, value: f64) -> Option<f64> {
+ assert!(!value.is_nan(), "value must not be NaN");
+ self.view().rank(value)
+ }
+
+ /// Compute approximate quantile value corresponding to the given normalized rank.
+ ///
+ /// Returns `None` if TDigest is empty.
+ ///
+ /// # Panics
+ ///
+ /// If rank is not in [0.0, 1.0].
+ pub fn quantile(&self, rank: f64) -> Option<f64> {
+ assert!((0.0..=1.0).contains(&rank), "rank must be in [0.0, 1.0]");
+ self.view().quantile(rank)
+ }
+
+ /// Converts this immutable TDigest into a mutable one.
+ pub fn into_mut(mut self) -> TDigestMut {
Review Comment:
To be symmetry with `TDigestMut::freeze`. I change it to `unfreeze` now. But I find it somewhat rare :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622259722
##########
src/tdigest/serialization.rs:
##########
@@ -17,6 +41,45 @@ impl TDigest {
});
bytes.push(SERIAL_VERSION);
bytes.push(TDIGEST_FAMILY_ID);
+ LE::write_u16(&mut bytes, self.k);
+ bytes.push({
+ let mut flags = 0;
+ if self.is_empty() {
+ flags |= FLAGS_IS_EMPTY;
+ }
+ if self.is_single_value() {
+ flags |= FLAGS_IS_SINGLE_VALUE;
+ }
+ if self.reverse_merge {
+ flags |= FLAGS_REVERSE_MERGE;
+ }
+ flags
+ });
+ LE::write_u16(&mut bytes, 0); // unused
+ if self.is_empty() {
+ return bytes;
+ }
+ if self.is_single_value() {
+ LE::write_f64(&mut bytes, self.min);
+ return bytes;
+ }
+ LE::write_u32(&mut bytes, self.centroids.len() as u32);
Review Comment:
Not sure if this can result in data loss. Seems possible if `self.k = u16::MAX` and thus the largets cap is `u16::MAX * 2 + 10 > u32::MAX`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on code in PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#discussion_r2622046267
##########
src/tdigest/sketch.rs:
##########
@@ -0,0 +1,445 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::convert::identity;
+
+const BUFFER_MULTIPLIER: usize = 4;
+
+/// T-Digest sketch for estimating quantiles and ranks.
+///
+/// See the [module documentation](super) for more details.
+#[derive(Debug, Clone, PartialEq)]
+pub struct TDigest {
+ k: usize,
+
+ reverse_merge: bool,
+ min: f64,
+ max: f64,
+
+ centroids: Vec<Centroid>,
+ centroids_weight: u64,
+ centroids_capacity: usize,
+ buffer: Vec<f64>,
+}
+
+impl Default for TDigest {
+ fn default() -> Self {
+ TDigest::new(Self::DEFAULT_K)
+ }
+}
+
+impl TDigest {
+ /// The default value of K if one is not specified.
+ pub const DEFAULT_K: usize = 200;
+
+ /// Creates a tdigest instance with the given value of k.
+ ///
+ /// # Panics
+ ///
+ /// If k is less than 10
+ pub fn new(k: usize) -> Self {
+ assert!(k >= 10, "k must be at least 10");
+
+ let fudge = if k < 30 { 30 } else { 10 };
+ let centroids_capacity = (k * 2) + fudge;
+
+ let centroids = Vec::with_capacity(centroids_capacity);
+ let buffer = Vec::with_capacity(centroids_capacity * BUFFER_MULTIPLIER);
Review Comment:
Never overflow. Perhaps consider using `Box<[T]>` for a never realloc variant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3659120900
cc @notfilippo @freakyzoidberg
cc @leerho based on https://github.com/apache/datasketches-rust/issues/2#issuecomment-3657395474, I made this impl as a combination of C++ & Java version. Welcome to give it a review. I'll add serde support today or tomorrow. But I'm still not quite sure what CDF and PMF are. It's possible to convey C++'s impl as is but I wonder a real world use case to understand its definition and usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] feat: implement T-Digest (datasketches-rust)
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #23:
URL: https://github.com/apache/datasketches-rust/pull/23#issuecomment-3659112341
Basic functions are ready. Anyone can drop a review now.
Other features may or may not included in this PR and they will be new commits if any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org