You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/08/21 21:57:03 UTC

[PR] Give the funnel functions the query's null handling option [pinot]

Jackie-Jiang opened a new pull request, #19332:
URL: https://github.com/apache/pinot/pull/19332

   ## Summary
   
   Part of #19218. Follows #19316 (sketch-backed distinct counts) and #19326 (value aggregations).
   
   The six funnel types — `FUNNELMAXSTEP`, `FUNNELCOMPLETECOUNT`, `FUNNELSTEPDURATIONSTATS`, `FUNNELMATCHSTEP`,
   `FUNNELEVENTSFUNCTIONEVAL` and `FUNNELCOUNT` — never received the query's null handling option, so a null row was
   read as the column default whatever the query asked for. Both funnel architectures are covered:
   `FunnelBaseAggregationFunction` and its four subclasses, the standalone `FunnelEventsFunctionEval`, and
   `FUNNELCOUNT`, whose aggregation lives in `AggregationStrategy` and its three strategy subclasses.
   
   ### Which null disqualifies a row is decided per column
   
   A funnel reads more than one column, so it cannot extend `NullableSingleInputAggregationFunction`. It follows the
   multi-input precedent set by `CovarianceAggregationFunction` — holding the flag itself and skipping rows through a
   local helper — and each column's rule follows from what that column is *for*:
   
   | Column | Rule | Why |
   |---|---|---|
   | Timestamp (window funnels) | skip the row | an aggregate ignores rows whose input is NULL, and the event has no position in the window |
   | `correlateBy` key (`FUNNELCOUNT`) | skip the row | the key is what distinct counts are taken over; with a composite key a null in any component leaves the whole key undefined |
   | Step predicates | **not** gated | a predicate over NULL is UNKNOWN, which SQL treats as not satisfied wherever a boolean is consumed, so a null step already means that step did not match |
   | Extra fields | **not** gated | payload on a matched event, not an input; dropping the row would lose an event that really happened |
   
   There is no SQL standard for funnel functions themselves, but each column role has one, and this is where they land.
   
   ### Both were real defects
   
   `BaseTransformFunction#getNullBitmap` ORs its arguments' bitmaps, so null timestamps genuinely arrive. One was
   being read as the column default and planted a step event at a **fabricated time**, corrupting window ordering and
   able to manufacture funnel completions. A null `correlateBy` key was read as the default's **dictionary id — a real
   user's id** — so the funnel credited that user with a step they never took.
   
   ### Testing
   
   `FunnelNullHandlingTest` is new and covers both architectures, including that `KEEP_ALL` must not fabricate a dummy
   event for a null-timestamp row. Every case was verified by mutation rather than by inspection.
   
   `SyntheticBlockValSets` gains a single-value dictionary-ids fixture, and its class comment now records that the
   null bitmap is independent of the values — which is why a test wanting "nothing was aggregated" in the disabled
   mode needs an untouched holder rather than an all-null block.
   
   `AggregationFunctionNullContractTest` gains multi-value block shapes, which make `SUMARRAYLONG` and `SUMARRAYDOUBLE`
   drivable by the generic harness for the first time — the set of undrivable functions shrinking rather than growing —
   and a pin on the funnels' empty answer, which is zero rather than `NULL` and was previously unpinned.
   
   ### Contract documentation
   
   With this the option reaches every user-facing aggregation, so the first known deviation on `AggregationFunction`
   is removed. It is replaced by what the series settled: how a multi-input function decides which nulls matter, and
   the three functions that stay outside the contract (`ParentExprMinMax`, `ChildExprMinMax`, `TimeSeriesAggregation`
   — none of them an aggregate a query can name). The multi-stage engine's unconditional null handling remains as the
   sole outstanding deviation.
   
   ### Known, deliberately not fixed here
   
   `FunnelEventsFunctionEval#getExtraFieldsBlocks` and `TimeSeriesAggregationFunction` switch on the **logical** value
   type rather than the stored type, so `BOOLEAN` and `JSON` columns are rejected where they would work. That is a
   type-dispatch bug rather than a null-handling one, and it *widens* accepted input, so it is going in its own PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Give the funnel functions the query's null handling option [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19332:
URL: https://github.com/apache/pinot/pull/19332#issuecomment-5376250039

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19332?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `47.78157%` with `153 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 57.77%. Comparing base ([`3c92923`](https://app.codecov.io/gh/apache/pinot/commit/3c92923b82d0ee75ecaa863d07bf7f9356ec2e35?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e56d88c`](https://app.codecov.io/gh/apache/pinot/commit/e56d88ccca1a868ed7f9fff9f87ee5e8c0381425?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19332?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...w/FunnelEventsFunctionEvalAggregationFunction.java](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fquery%2Faggregation%2Ffunction%2Ffunnel%2Fwindow%2FFunnelEventsFunctionEvalAggregationFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9mdW5uZWwvd2luZG93L0Z1bm5lbEV2ZW50c0Z1bmN0aW9uRXZhbEFnZ3JlZ2F0aW9uRnVuY3Rpb24uamF2YQ==) | 6.34% | [59 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...egation/function/HistogramAggregationFunction.java](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fquery%2Faggregation%2Ffunction%2FHistogramAggregationFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9IaXN0b2dyYW1BZ2dyZWdhdGlvbkZ1bmN0aW9uLmphdmE=) | 46.98% | [42 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...gregation/function/funnel/AggregationStrategy.java](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fquery%2Faggregation%2Ffunction%2Ffunnel%2FAggregationStrategy.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9mdW5uZWwvQWdncmVnYXRpb25TdHJhdGVneS5qYXZh) | 42.25% | [38 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...n/funnel/window/FunnelBaseAggregationFunction.java](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fquery%2Faggregation%2Ffunction%2Ffunnel%2Fwindow%2FFunnelBaseAggregationFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9mdW5uZWwvd2luZG93L0Z1bm5lbEJhc2VBZ2dyZWdhdGlvbkZ1bmN0aW9uLmphdmE=) | 84.48% | [2 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19332?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (3c92923) and HEAD (e56d88c). Click for more details.
   > 
   > <details><summary>HEAD has 24 uploads less than BASE</summary>
   >
   >| Flag | BASE (3c92923) | HEAD (e56d88c) |
   >|------|------|------|
   >|unittests|2|1|
   >|java-25|6|1|
   >|temurin|6|1|
   >|lane-a|2|0|
   >|integration|4|0|
   >|integration1|2|0|
   >|lane-b|2|0|
   >|integration2|2|0|
   >|unittests2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19332      +/-   ##
   ============================================
   - Coverage     66.97%   57.77%   -9.21%     
   + Complexity     1424        1    -1423     
   ============================================
     Files          3463     2660     -803     
     Lines        222078   162062   -60016     
     Branches      34957    26215    -8742     
   ============================================
   - Hits         148743    93634   -55109     
   + Misses        61467    60480     -987     
   + Partials      11868     7948    -3920     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.77% <47.78%> (-9.21%)` | :arrow_down: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.77% <47.78%> (-9.21%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.77% <47.78%> (-9.20%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.77% <47.78%> (+0.08%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19332/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19332?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Give the funnel functions the query's null handling option [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #19332:
URL: https://github.com/apache/pinot/pull/19332


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Give the funnel functions the query's null handling option [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19332:
URL: https://github.com/apache/pinot/pull/19332#issuecomment-5394047570

   Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/1002


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org