You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/09/02 23:57:30 UTC

[PR] Make filter count and bitmap shortcuts three-valued under null handling [pinot]

Jackie-Jiang opened a new pull request, #19440:
URL: https://github.com/apache/pinot/pull/19440

## Summary

With null handling enabled, a null row is UNKNOWN to a predicate on its column: neither the predicate nor its negation selects it. Several filter paths were still two-valued and let such rows through. All of them are fixed here by making the bitmap shortcut layer three-valued rather than by declining shortcuts, so no null-handling query gets slower and the null-handling-disabled path is unchanged.

### What was wrong
- **Always-true / always-false predicates.** An evaluator's verdict is over the dictionary's real values. The leaf collapsed to `MatchAllFilterOperator` / `EmptyFilterOperator` (or a two-valued bitmap), so `c <> 99999` selected the null rows and `NOT (c = 99999)` selected them too. `StarTreeUtils` dropped such predicates from its map, so the column escaped the null check that decides whether a star-tree may serve the query, and the star-tree answered with the null row folded into the column's default value.
- **Index-based shortcuts.** `InvertedIndexFilterOperator`, `SortedIndexBasedFilterOperator` and `RangeIndexBasedFilterOperator` computed `getNumMatchingDocs()` and `getBitmaps()` from postings alone, while their `getTrues()` excluded the null vector. `COUNT(*)` on its own takes the fast filtered count whenever the filter can count, so `SELECT COUNT(*) FROM t WHERE c <> 1` counted the null rows and `WHERE c = <default>` matched them. Index-based `DISTINCT` and the vector pre-filter consumed the same bitmaps.
- **Negation and composition.** `NotFilterOperator` counted `numDocs - child` and inverted the child's bitmaps as if no document were UNKNOWN, and NOT / AND / OR reported no UNKNOWN documents through `getNulls()`, so a nested boolean tree under a negation treated UNKNOWN rows as false on the iterator path.

### What changes
- `BitmapCollection` gains an optional null bitmap via `excludingNulls`. True documents are the union (or its complement when inverted) minus the nulls; `invert()` keeps the null bitmap since NOT of UNKNOWN is UNKNOWN; `reduce()`, `andCardinality`, `orCardinality` and the new `getCardinality()` honor it. The two-valued fast paths are untouched when no null bitmap is present.
- `BaseColumnFilterOperator` reads the column's null bitmap once at construction, only when null handling is enabled and the vector is non-empty. `getTrues()`, `getNulls()`, `mayHaveNulls()` and the leaves' shortcuts share it, so a consuming segment's vector (which hands out a copy per read) is copied once and every view of the operator sees the same rows.
- The inverted, sorted and range leaves and `BitmapBasedFilterOperator` attach that bitmap to their collection and count through it when nulls exist. The no-null path is the previous code.
- `BaseFilterOperator.mayHaveNulls()` is a metadata-only, possibly pessimistic answer. `NotFilterOperator` counts through the child's inverted bitmaps only when the child may have nulls, and propagates its child's `getNulls()`. `AndFilterOperator` / `OrFilterOperator` compute their own null bitmap when a child has one and implement `getNulls()` as "neither true nor false".
- `FilterOperatorUtils.getLeafFilterOperator` builds a three-valued `BitmapBasedFilterOperator` for an always-true / always-false predicate on a column with nulls, instead of a constant. `StarTreeUtils` keeps such predicates in its map so the null check sees the column. `FilterOperatorUtils.getNullBitmap` / `hasNulls` are the shared helpers.
- `PredicateEvaluatorProvider` throws `IllegalStateException` rather than `UnsupportedOperationException` for an unsupported predicate type.

### Behavior change
Query answers change only with null handling enabled, and only where they were wrong: counts, distinct values and star-tree aggregates no longer include null rows that the predicate or its negation cannot select. Leaves that never model nulls (text, JSON, H3, vector) are unchanged.

### Tests
`BitmapCollectionTest` covers the null cases including inversion. `NotFilterOperatorTest`, `AndFilterOperatorTest` and `OrFilterOperatorTest` check count, bitmaps, `getNulls()` and the iterator path agree under nulls, including nested trees under a negation. `InvertedIndexFilterOperatorTest` mocks a null vector on both sides of exclusive. `NullHandlingEnabledQueriesTest` runs `COUNT(*)` with `<>`, equality on the stored default, `NOT`, `IN` and ranges over inverted, sorted and range-indexed columns, a multi-value column, a consuming segment, and index-based `DISTINCT`. `StarTreeNullHandlingQueriesTest` checks the star-tree is refused for always-true / negated always-false predicates and OR clauses on a nullable column and kept on a non-nullable one.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org