You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2025/11/11 10:19:56 UTC
[PR] NoDict: remove HLL++ inflation; tests: relax cardinality tolerance [pinot]
xiangfu0 opened a new pull request, #17186:
URL: https://github.com/apache/pinot/pull/17186
HLL++ is probabilistic and may under/over-estimate.
The randomized test now allows 10% or ≥1 deviation between approximate and exact cardinality.
Removed the prior 10% upward bias in NoDictColumnStatisticsCollector.getCardinality() to avoid systematic overestimation and reduce flakiness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517203168
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -208,8 +222,17 @@ public int getMaxRowLengthInBytes() {
@Override
public int getCardinality() {
- // Get approximate distinct count estimate using HLL++
- // Increase by 10% to increase probability of not returning lower than actual cardinality
+ // If we are still tracking exact uniques, return exact cardinality
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid races
+ if (exact != null) {
+ int size = exact.size();
+ // Defense-in-depth: if exact tracking was concurrently disabled after we took the snapshot,
+ // prefer falling back to the HLL++ estimate to avoid any theoretical visibility concerns.
+ if (_exactUniquesRef.get() == exact) {
+ return size;
+ }
Review Comment:
The double-check pattern here is incorrect. After calling `exact.size()` on line 228, the size could become stale if another thread exceeds the threshold and nulls out the reference. The second check on line 231 doesn't prevent returning a stale size. Consider reading the size only after the second reference check, or accept that exact cardinality might slightly exceed the threshold in race conditions.
```suggestion
Set<Object> exact = _exactUniquesRef.get();
if (exact != null) {
// Only read the size after confirming the reference is still valid
// This avoids returning a stale size if the reference was nulled out concurrently
return exact.size();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [flaky test]NoDict: remove HLL++ inflation; tests: relax cardinality tolerance [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2513679122
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollectorTest.java:
##########
@@ -781,9 +781,13 @@ private void compareCollectorStats(NoDictColumnStatisticsCollector noDictCollect
String context = String.format("Iteration %d, DataType: %s, SingleValue: %s, Data: %s",
iteration, dataType, isSingleValue, Arrays.deepToString(testData));
- assertTrue(noDictCollector.getCardinality() >= expectedCollector.getCardinality(),
- "Approx Cardinality " + noDictCollector.getCardinality() + " is lower than actual cardinality "
- + expectedCollector.getCardinality() + " for " + context);
+ int approx = noDictCollector.getCardinality();
+ int exact = expectedCollector.getCardinality();
+ // allow 10% or at least 1; guard against overflow/precision in intermediate calculation
+ int tolerance = Math.max(1, (int) Math.min(Integer.MAX_VALUE, Math.ceil(0.1 * (double) exact)));
Review Comment:
The `Math.min(Integer.MAX_VALUE, ...)` check is unnecessary because `Math.ceil(0.1 * (double) exact)` cannot exceed `Integer.MAX_VALUE` given that `exact` is an `int`. The result of `0.1 * exact` will always be within representable double range, and casting to int will clamp any theoretical overflow. Consider simplifying to `Math.max(1, (int) Math.ceil(0.1 * exact))`.
```suggestion
// allow 10% or at least 1
int tolerance = Math.max(1, (int) Math.ceil(0.1 * exact));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517346783
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +242,40 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid check-then-act race
+ if (exact == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
+ } else if (value instanceof BigDecimal) {
+ // Use string representation to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
+ key = ((BigDecimal) value).toString();
+ } else {
+ key = value;
+ }
+ exact.add(key);
+ if (exact.size() > EXACT_UNIQUE_TRACKING_THRESHOLD) {
Review Comment:
Race condition between `add()` and `size()` check can cause the set to grow unbounded beyond the threshold. Multiple threads could simultaneously add elements and check size when it's exactly at the threshold, with each seeing size <= threshold before their additions complete. Use the return value from `add()` (which returns true if element was added) to increment an `AtomicInteger` counter, and check the counter instead of set size to ensure deterministic threshold enforcement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517089823
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -52,6 +54,10 @@ public class NoDictColumnStatisticsCollector extends AbstractColumnStatisticsCol
private boolean _sealed = false;
// HLL Plus generally returns approximate cardinality >= actual cardinality which is desired
private final HyperLogLogPlus _hllPlus;
+ // Track exact uniques up to a threshold to avoid small-N underestimation and test flakiness
+ private static final int EXACT_UNIQUE_TRACKING_THRESHOLD = 2048;
+ // Volatile because this reference can be nulled out once the threshold is exceeded
+ private volatile Set<Object> _exactUniques = new HashSet<>();
Review Comment:
The `volatile` keyword on the `Set` reference doesn't provide thread-safety for the `HashSet` operations themselves. Concurrent modifications to the `HashSet` (via `add()`) can lead to race conditions. Consider using `Collections.synchronizedSet()` or `ConcurrentHashMap.newKeySet()` if this collector can be accessed by multiple threads.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +238,34 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ if (_exactUniques == null) {
+ return;
+ }
Review Comment:
Reading `_exactUniques` and then calling `add()` on it creates a check-then-act race condition. Another thread could null out `_exactUniques` between the null check and the `add()` operation, causing a `NullPointerException`. Store the reference in a local variable at the start of the method to ensure consistent access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517101707
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -52,6 +54,10 @@ public class NoDictColumnStatisticsCollector extends AbstractColumnStatisticsCol
private boolean _sealed = false;
// HLL Plus generally returns approximate cardinality >= actual cardinality which is desired
private final HyperLogLogPlus _hllPlus;
+ // Track exact uniques up to a threshold to avoid small-N underestimation and test flakiness
+ private static final int EXACT_UNIQUE_TRACKING_THRESHOLD = 2048;
+ // Volatile because this reference can be nulled out once the threshold is exceeded
+ private volatile Set<Object> _exactUniques = ConcurrentHashMap.newKeySet();
Review Comment:
Using `volatile` for `_exactUniques` is insufficient for thread-safe access. The `getCardinality()` method reads this field and calls `size()` on the set, but if another thread nulls out `_exactUniques` between the null check and the `size()` call, a `NullPointerException` could occur. Consider using `AtomicReference<Set<Object>>` or synchronized blocks to ensure thread-safe access to both the reference and the set operations.
```suggestion
// Use a lock object to synchronize access to _exactUniques for thread safety
private final Object _exactUniquesLock = new Object();
private Set<Object> _exactUniques = ConcurrentHashMap.newKeySet();
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +239,35 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ Set<Object> exact = _exactUniques; // local snapshot to avoid check-then-act race
+ if (exact == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
+ } else if (value instanceof BigDecimal) {
+ // Use string representation to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
+ key = ((BigDecimal) value).toString();
+ } else {
+ key = value;
+ }
+ exact.add(key);
+ if (exact.size() > EXACT_UNIQUE_TRACKING_THRESHOLD) {
+ // Drop exact tracking once the threshold is exceeded to avoid unbounded memory
+ _exactUniques = null;
+ }
Review Comment:
Race condition: multiple threads can simultaneously check `exact.size() > EXACT_UNIQUE_TRACKING_THRESHOLD` and all attempt to null out `_exactUniques`. Additionally, other threads might still be adding to the set after it's been nulled, leading to continued memory growth beyond the threshold. Use `AtomicReference.compareAndSet()` or synchronization to ensure exactly one thread performs the nullification and that no further additions occur after the threshold is exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2516963829
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -229,4 +243,24 @@ private void updateHllPlus(Object value) {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ if (_exactUniques == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
Review Comment:
The code references `ByteArray` class without importing it. Add the required import statement for `ByteArray`, likely from `org.apache.pinot.spi.utils.ByteArray` or equivalent package.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -74,6 +79,7 @@ public void collect(Object entry) {
throw new UnsupportedOperationException();
}
updateMinMax(value);
+ trackExactUnique(value);
Review Comment:
The `trackExactUnique()` method is called before `updateHllPlus()` in the single-value path, but after the threshold is exceeded, `_exactUniques` will be null while HLL++ continues to track values. Consider reordering to call `updateHllPlus()` first to ensure HLL++ state is consistent even when exact tracking is active, or document why the current ordering is intentional.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -229,4 +243,24 @@ private void updateHllPlus(Object value) {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ if (_exactUniques == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
+ } else if (value instanceof BigDecimal) {
+ // Use string representation to avoid scale-related equality issues
Review Comment:
[nitpick] The comment mentions 'scale-related equality issues' but doesn't explain what specific problem this solves. Consider expanding the comment to clarify that BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default, and string representation normalizes this.
```suggestion
// Use string representation to avoid scale-related equality issues:
// BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
// but their string representations normalize the value for cardinality tracking.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [flaky test]NoDict: remove HLL++ inflation; tests: relax cardinality tolerance [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17186:
URL: https://github.com/apache/pinot/pull/17186#issuecomment-3516379362
### :x: 11 Tests Failed:
| Tests completed | Failed | Passed | Skipped |
|---|---|---|---|
| 4782 | 11 | 4771 | 0 |
<details><summary>View the top 3 failed test(s) by shortest run time</summary>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testForwardIndexHandlerChangeCompression</code></pre>
> <details><summary>Stack Traces | 1.18s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testSimpleEnableDictionaryMV</code></pre>
> <details><summary>Stack Traces | 1.2s run time</summary>
>
> > <pre><code class="language-python">expected [20516] but found [18651]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testSimpleEnableDictionarySV</code></pre>
> <details><summary>Stack Traces | 1.2s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testEnableTextIndexOnExistingRawColumn[v1](1)</code></pre>
> <details><summary>Stack Traces | 1.31s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testEnableTextIndexOnExistingRawColumn[v3](2)</code></pre>
> <details><summary>Stack Traces | 1.33s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testEnableTextIndexOnExistingRawColumn</code></pre>
> <details><summary>Stack Traces | 1.34s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testEnableDictAndOtherIndexesMV</code></pre>
> <details><summary>Stack Traces | 2.69s run time</summary>
>
> > <pre><code class="language-python">expected [20516] but found [18651]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testUpdateDefaultColumns[v3](2)</code></pre>
> <details><summary>Stack Traces | 2.97s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testUpdateDefaultColumns</code></pre>
> <details><summary>Stack Traces | 2.98s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testEnableDictAndOtherIndexesSV</code></pre>
> <details><summary>Stack Traces | 3.2s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
> <pre><code class="language-python">org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessorTest::testUpdateDefaultColumns[v1](1)</code></pre>
> <details><summary>Stack Traces | 3.85s run time</summary>
>
> > <pre><code class="language-python">expected [6] but found [5]</code></pre>
>
> </details>
</details>
To view more test analytics, go to the [Test Analytics Dashboard](https://app.codecov.io/gh/apache/pinot/tests/xiangfu0%3Afix%2Fhll-cardinality-tolerance)
<sub>📋 Got 3 mins? [Take this short survey](https://forms.gle/22i53Qa1CySZjA6c7) to help us improve Test Analytics.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517082227
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -52,6 +54,9 @@ public class NoDictColumnStatisticsCollector extends AbstractColumnStatisticsCol
private boolean _sealed = false;
// HLL Plus generally returns approximate cardinality >= actual cardinality which is desired
private final HyperLogLogPlus _hllPlus;
+ // Track exact uniques up to a threshold to avoid small-N underestimation and test flakiness
+ private static final int EXACT_UNIQUE_TRACKING_THRESHOLD = 2048;
+ private Set<Object> _exactUniques = new HashSet<>();
Review Comment:
The `_exactUniques` field is reassigned to `null` after exceeding the threshold, but it's not marked as `volatile` or synchronized. If this class is used in a concurrent context, this pattern could lead to visibility issues. Consider documenting thread-safety assumptions or using `volatile` if concurrent access is expected.
```suggestion
private volatile Set<Object> _exactUniques = new HashSet<>();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2521301969
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +242,40 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid check-then-act race
+ if (exact == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
+ } else if (value instanceof BigDecimal) {
+ // Use string representation to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
+ key = ((BigDecimal) value).toString();
+ } else {
+ key = value;
+ }
+ exact.add(key);
+ if (exact.size() > EXACT_UNIQUE_TRACKING_THRESHOLD) {
+ // Drop exact tracking once the threshold is exceeded to avoid unbounded memory.
+ // Note on concurrency: we intentionally operate on the local snapshot 'exact'.
+ // Another thread can successfully CAS the AtomicReference to null while this thread still holds
+ // the snapshot. That is safe: the snapshot remains valid for this call, and any additional adds
+ // are bounded and the snapshot becomes unreachable after this method returns (eligible for GC).
+ // Using CAS ensures only one thread disables tracking.
+ _exactUniquesRef.compareAndSet(exact, null);
Review Comment:
yea we don't need this. all our stats collectors are used within the bounds of a single thread and aren't thread safe.
we can remove this overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [flaky test]NoDict: remove HLL++ inflation; tests: relax cardinality tolerance [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on PR #17186:
URL: https://github.com/apache/pinot/pull/17186#issuecomment-3516927047
@xiangfu0 We added 10% overestimation to prevent unknown issues if approx cardinality < actual cardinality.
HLL has an upper bound of error so adding 10% buffer guarantees that approx cardinality > actual cardinality when unique elements > threshold.
For flakiness in tests when unique elements < threshold, I plan to store unique values upto a threshold so that we get actual cardinality in such cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [flaky test]NoDict: remove HLL++ inflation; tests: relax cardinality tolerance [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2513653611
##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollectorTest.java:
##########
@@ -781,9 +781,12 @@ private void compareCollectorStats(NoDictColumnStatisticsCollector noDictCollect
String context = String.format("Iteration %d, DataType: %s, SingleValue: %s, Data: %s",
iteration, dataType, isSingleValue, Arrays.deepToString(testData));
- assertTrue(noDictCollector.getCardinality() >= expectedCollector.getCardinality(),
- "Approx Cardinality " + noDictCollector.getCardinality() + " is lower than actual cardinality "
- + expectedCollector.getCardinality() + " for " + context);
+ int approx = noDictCollector.getCardinality();
+ int exact = expectedCollector.getCardinality();
+ int tolerance = Math.max(1, (int) Math.ceil(0.1 * exact)); // allow 10% or at least 1
Review Comment:
The tolerance calculation could overflow when `exact` is very large. For example, if `exact` is close to `Integer.MAX_VALUE`, multiplying by 0.1 and casting could produce unexpected results. Consider using `(int) Math.min(Integer.MAX_VALUE, Math.ceil(0.1 * exact))` or handling the calculation in long arithmetic before casting.
```suggestion
int tolerance = Math.max(1, (int) Math.min(Integer.MAX_VALUE, Math.ceil(0.1 * exact))); // allow 10% or at least 1, prevent overflow
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517187574
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -208,8 +222,12 @@ public int getMaxRowLengthInBytes() {
@Override
public int getCardinality() {
- // Get approximate distinct count estimate using HLL++
- // Increase by 10% to increase probability of not returning lower than actual cardinality
+ // If we are still tracking exact uniques, return exact cardinality
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid races
+ if (exact != null) {
+ return exact.size();
Review Comment:
The exact set can become null between the `get()` and `size()` calls due to concurrent modification in `trackExactUnique()`. Although unlikely in practice due to the timing window, this could result in a `NullPointerException`. Consider using a local variable pattern with a null check after reading the set reference.
```suggestion
// Defensive: exact can become null between get() and size() due to concurrent modification
return exact != null ? exact.size() : (int) Math.round(_hllPlus.cardinality() * 1.1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2518905386
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +242,40 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid check-then-act race
+ if (exact == null) {
+ return;
+ }
+ Object key;
+ if (value instanceof byte[]) {
+ key = new ByteArray((byte[]) value);
+ } else if (value instanceof BigDecimal) {
+ // Use string representation to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
+ key = ((BigDecimal) value).toString();
+ } else {
+ key = value;
+ }
+ exact.add(key);
+ if (exact.size() > EXACT_UNIQUE_TRACKING_THRESHOLD) {
+ // Drop exact tracking once the threshold is exceeded to avoid unbounded memory.
+ // Note on concurrency: we intentionally operate on the local snapshot 'exact'.
+ // Another thread can successfully CAS the AtomicReference to null while this thread still holds
+ // the snapshot. That is safe: the snapshot remains valid for this call, and any additional adds
+ // are bounded and the snapshot becomes unreachable after this method returns (eligible for GC).
+ // Using CAS ensures only one thread disables tracking.
+ _exactUniquesRef.compareAndSet(exact, null);
Review Comment:
Why is any of this needed? What exactly is the threading model and potential for contention or race conditions here?
cc - @krishan1390
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #17186:
URL: https://github.com/apache/pinot/pull/17186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [enhancement]NoDict: hybrid cardinality (exact ≤ 2,048; HLL++ with +10% buffer) [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17186:
URL: https://github.com/apache/pinot/pull/17186#discussion_r2517235151
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/stats/NoDictColumnStatisticsCollector.java:
##########
@@ -223,10 +242,35 @@ public void seal() {
private void updateHllPlus(Object value) {
if (value instanceof BigDecimal) {
- // Canonicalize BigDecimal as string to avoid scale-related equality issues
+ // Canonicalize BigDecimal as string to avoid scale-related equality issues:
+ // BigDecimals with different scales (e.g., 1.0 vs 1.00) are not equal by default,
+ // but their string representations normalize the value for cardinality tracking.
_hllPlus.offer(((BigDecimal) value).toString());
} else {
_hllPlus.offer(value);
}
}
+
+ private void trackExactUnique(Object value) {
+ Set<Object> exact = _exactUniquesRef.get(); // local snapshot to avoid check-then-act race
+ if (exact == null) {
+ return;
+ }
Review Comment:
After checking `exact != null`, another thread could execute `compareAndSet(exact, null)` on line 273 before `exact.add(key)` on line 270, causing a NullPointerException. While the local snapshot prevents the check-then-act race on the reference itself, it doesn't protect against the set being nulled out by another thread's `compareAndSet` after this method captures the reference. Consider adding a try-catch around `exact.add(key)` to handle concurrent nullification gracefully, or document that this is acceptable behavior (failed adds after threshold are benign).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org