You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "davecromberge (via GitHub)" <gi...@apache.org> on 2026/09/03 10:50:01 UTC
[PR] Add CpcUnion.update(MemorySegment) (datasketches-java)
davecromberge opened a new pull request, #759:
URL: https://github.com/apache/datasketches-java/pull/759
## What
`CpcUnion` only accepts a `CpcSketch`, so merging a stored sketch means deserializing it first:
```java
union.update(CpcSketch.heapify(MemorySegment.ofArray(bytes)));
```
That builds an entire sketch — sliding window, pair table, HIP registers — which the union then walks once and discards. This adds an overload that skips it:
```java
union.update(MemorySegment.ofArray(bytes));
```
Where the union already holds a bit matrix, the image's coupons are decoded straight into it. Sparse, Hybrid and Pinned images decode directly. Sliding partly inverts its logic — a coupon can be signalled by the *absence* of a pair — so it still goes through a sketch. Either way the resulting union is byte-identical.
## Use case
Unioning columns of serialized sketches, which is what query engines and offline rollup jobs mostly do. In Pinot this path is taken by `distinctCountRawCpcSketch`, by segment rollup, and by star-tree index construction; all three heapify every input today.
## Measured
New characterization profile `CpcUnionDeserializeSpeedProfile` — 32 stored sketches merged per trial at lgK=12 — median of 4 interleaved runs of each entry point against the same jar.
At lgK=12 each of the 32 sketches holds `uniques/32`, so the lower half of the table decodes directly and the upper half is Sliding, which falls back to a sketch.
| uniques | heapify ns/sketch | segment ns/sketch | change |
| --- | --- | --- | --- |
| 1024 | 880 | 722 | −18.0% |
| 2048 | 1390 | 949 | −31.7% |
| 4096 | 2370 | 1313 | −44.6% |
| 8192 | 4261 | 1975 | −53.7% |
| 16384 | 9676 | 3371 | −65.2% |
| 32768 | 8719 | 6466 | −25.8% |
| 65536 | 15545 | 12926 | −16.9% |
| 262144 | 15807 | 15195 | −3.9% |
## Notes
- Adds one public method; nothing existing changes behaviour.
- `uncompressTheWindow` now returns the window instead of assigning it into a target sketch, matching `uncompressTheSurprisingValues` beside it, so both decode primitives can serve either caller.
- `CpcUnionSegmentUpdateTest` asserts both entry points produce byte-identical unions across every flavor, both union states, lgK reduction, and random input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] Add CpcUnion.update(MemorySegment) (datasketches-java)
Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on PR #759:
URL: https://github.com/apache/datasketches-java/pull/759#issuecomment-5576966389
Hi Dave, as I recall, we do something similar in theta sketches, so this is a good idea. Would it be possible to check if we have this optimization in C++? If not we need to put it on a to-do list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] Add CpcUnion.update(MemorySegment) (datasketches-java)
Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho merged PR #759:
URL: https://github.com/apache/datasketches-java/pull/759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org