You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/01 16:38:21 UTC
(datafusion) branch gh-readonly-queue/main/pr-25744-37744ba97f1c1a8d8ab5d1ae779329d911cd7da7 created (now 059aaa217a)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-25744-37744ba97f1c1a8d8ab5d1ae779329d911cd7da7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 059aaa217a fix: clamp cosine_distance to enforce metric non-negativity and boundedness [0.0, 2.0] (#25744)
This branch includes the following new commits:
new 059aaa217a fix: clamp cosine_distance to enforce metric non-negativity and boundedness [0.0, 2.0] (#25744)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: clamp cosine_distance to enforce metric non-negativity and boundedness [0.0, 2.0] (#25744)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25744-37744ba97f1c1a8d8ab5d1ae779329d911cd7da7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 059aaa217abf2abea87af2afb1c41239ffcea265
Author: Ignatius Pang <23...@users.noreply.github.com>
AuthorDate: Thu Oct 1 16:37:57 2026 +0000
fix: clamp cosine_distance to enforce metric non-negativity and boundedness [0.0, 2.0] (#25744)
# Pull Request: Metric Non-Negativity and Boundedness in
`cosine_distance`
- **Target Repository**: `apache/datafusion` (`main`)
- **Fork Branch**:
`APAF-bioinformatics/datafusion:fix-cosine-distance-non-negative`
- **Commit**: `4ecba5e52a19c0ec1398e074632288b0666d5c28`
- **PR Comparison Link**:
[https://github.com/apache/datafusion/compare/main...APAF-bioinformatics:datafusion:fix-cosine-distance-non-negative](https://github.com/apache/datafusion/compare/main...APAF-bioinformatics:datafusion:fix-cosine-distance-non-negative)
## Rationale for this change
In `datafusion-functions-nested::cosine_distance`, the distance is
evaluated as:
```rust
builder.append_value(1.0 - dot / (sq1.sqrt() * sq2.sqrt()));
```
In IEEE-754 floating-point arithmetic, $\sqrt{S} \times \sqrt{S}$ does
not always equal $S$. For example, with $S = 3.0$ (such as when
comparing the identical unit vectors `[1.0, 1.0, 1.0]`):
- `dot = 3.0`
- `sq1 = 3.0`, `sq2 = 3.0`
- `sq1.sqrt() * sq2.sqrt() = 2.9999999999999996 < 3.0`
- `dot / (sq1.sqrt() * sq2.sqrt()) = 1.0000000000000002 > 1.0`
- `1.0 - ratio = -2.220446049250313e-16 < 0.0`
This violates fundamental metric axioms:
1. **Metric non-negativity**: $d(x, y) \ge 0$ for all $x, y$.
2. **Identity of indiscernibles**: $d(x, x) = 0$.
3. **Distance boundedness**: $d(x, y) \in [0.0, 2.0]$ for cosine
distance.
For opposite vectors, floating-point roundoff can similarly produce
distances slightly exceeding `2.0` (e.g. `2.0000000000000004`).
## What changes are included in this PR?
1. **Cosine Similarity & Distance Clamping**:
- Clamps the computed similarity ratio `(dot / (sq1.sqrt() *
sq2.sqrt())).clamp(-1.0, 1.0)`.
- Clamps the resulting cosine distance `(1.0 - sim).clamp(0.0, 2.0)`.
- Preserves standard IEEE-754 `NaN` propagation (since `f64::clamp` on
`NaN` evaluates `NaN < min` and `NaN > max` to `false`, propagating
`NaN`).
2. **Comprehensive Regression Tests**:
- Unit tests in `cosine_distance.rs`:
- `test_cosine_distance_identical_vectors_metric_non_negative`: Verifies
that identical vectors prone to $\sqrt{3}^2$ roundoff return strictly
`0.0` and never negative numbers.
- `test_cosine_distance_opposite_vectors_bounded`: Verifies that
opposite vectors return `2.0` and never exceed `2.0`.
- Sqllogictest in `cosine_distance.slt`: Added query testing `select
cosine_distance([1.0, 1.0, 1.0], [1.0, 1.0, 1.0]);` returning `0`.
## What is the testing strategy for this PR?
- Full crate test suite pass for `datafusion-functions-nested` (155
tests total):
```bash
cargo test -p datafusion-functions-nested
```
Result: `153 passed (unittests), 2 passed (doctests), 0 failed`.
- Full sqllogictest suite pass for `cosine_distance.slt`:
```bash
cargo test -p datafusion-sqllogictest --test sqllogictests --
cosine_distance
```
Result: `Progress: 1/1 files completed (100%), 0 failed`.
## Are there any user-facing changes?
No breaking API changes. Bug fix only: `cosine_distance` will never
return negative values (such as `-2.22e-16`) on identical vectors or
values $> 2.0$ on opposite vectors due to floating-point roundoff.
```
---------
Co-authored-by: ignatiuspang <Ig...@users.noreply.git.com>
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
datafusion/functions-nested/src/cosine_distance.rs | 3 ++-
.../sqllogictest/test_files/cosine_distance.slt | 24 ++++++++++++++++++++++
2 files changed, 26 insertions(+), 1 deletion(-)
diff --git a/datafusion/functions-nested/src/cosine_distance.rs b/datafusion/functions-nested/src/cosine_distance.rs
index 1637b2cea1..11e8447fb8 100644
--- a/datafusion/functions-nested/src/cosine_distance.rs
+++ b/datafusion/functions-nested/src/cosine_distance.rs
@@ -226,7 +226,8 @@ fn general_cosine_distance<O: OffsetSizeTrait>(arrays: &[ArrayRef]) -> Result<Ar
if sq1 == 0.0 || sq2 == 0.0 {
builder.append_null();
} else {
- builder.append_value(1.0 - dot / (sq1.sqrt() * sq2.sqrt()));
+ let sim = (dot / (sq1.sqrt() * sq2.sqrt())).clamp(-1.0, 1.0);
+ builder.append_value((1.0 - sim).clamp(0.0, 2.0));
}
}
diff --git a/datafusion/sqllogictest/test_files/cosine_distance.slt b/datafusion/sqllogictest/test_files/cosine_distance.slt
index 12bc5621ed..97a7cbc67f 100644
--- a/datafusion/sqllogictest/test_files/cosine_distance.slt
+++ b/datafusion/sqllogictest/test_files/cosine_distance.slt
@@ -29,12 +29,36 @@ select cosine_distance([1.0, 2.0, 3.0], [1.0, 2.0, 3.0]);
----
0
+# Identical vectors susceptible to floating point sqrt roundoff: distance = 0.0
+query R
+select cosine_distance([1.0, 1.0, 1.0], [1.0, 1.0, 1.0]);
+----
+0
+
+# Identical vectors metric non-negativity: distance >= 0.0
+query B
+select cosine_distance([1.0, 1.0, 1.0], [1.0, 1.0, 1.0]) >= 0.0;
+----
+true
+
# Opposite vectors: distance = 2.0
query R
select cosine_distance([1.0, 0.0], [-1.0, 0.0]);
----
2
+# Opposite 3D vectors bounded: distance = 2.0
+query R
+select cosine_distance([1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]);
+----
+2
+
+# Opposite vectors metric boundedness: distance <= 2.0
+query B
+select cosine_distance([1.0, 1.0, 1.0], [-1.0, -1.0, -1.0]) <= 2.0;
+----
+true
+
# 45-degree angle: distance ≈ 0.293
query R
select round(cosine_distance([1.0, 0.0], [1.0, 1.0]), 3);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org