You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/04/05 06:22:35 UTC
[PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
BiteTheDDDDt opened a new pull request, #62126:
URL: https://github.com/apache/doris/pull/62126
pick from #62043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62126:
URL: https://github.com/apache/doris/pull/62126#issuecomment-4188383690
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #62126:
URL: https://github.com/apache/doris/pull/62126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62126:
URL: https://github.com/apache/doris/pull/62126#issuecomment-4188488088
# BE UT Coverage Report
Increment line coverage `95.40% (83/87)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e_5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e_5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.98% (19820/37409) |
| Line Coverage | 36.46% (186247/510832) |
| Region Coverage | 32.75% (144432/440983) |
| Branch Coverage | 33.92% (63384/186858) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62126:
URL: https://github.com/apache/doris/pull/62126#issuecomment-4188383650
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #62126:
URL: https://github.com/apache/doris/pull/62126#discussion_r3036483478
##########
be/src/exprs/aggregate/aggregate_function_window_funnel_v2.h:
##########
@@ -393,123 +393,209 @@ struct WindowFunnelStateV2 {
if (matched) {
events_timestamp[event_idx] = {events_timestamp[event_idx - 1].first_ts,
evt.timestamp, i};
- if (event_idx > curr_level) {
- curr_level = event_idx;
- }
+ curr_level = std::max(event_idx, curr_level);
if (event_idx + 1 == event_count) {
return event_count;
}
}
}
}
- if (curr_level + 1 > max_level) {
- max_level = curr_level + 1;
- }
+ max_level = std::max(curr_level + 1, max_level);
}
return max_level;
}
- /// DEDUPLICATION mode: if a previously matched event level appears again,
- /// the current chain is terminated and max_level is updated.
- /// This preserves V1 semantics where duplicate events break the chain.
+ /// DEDUPLICATION mode (multi-pass, matching V1 semantics):
+ ///
+ /// V1 tries each E0-matching row as a chain starting point and searches for
+ /// conditions one at a time. Before accepting a match for condition K, V1 checks
+ /// the "gap" (rows between the previous match and the current match) for any
+ /// re-occurrence of already-matched conditions 0..K-1. If found, the chain breaks.
+ ///
+ /// A single-pass approach cannot correctly implement this because V2's greedy
+ /// multi-condition-per-row advancement creates premature higher-level matches
+ /// that then conflict with later events. The multi-pass approach iterates over
+ /// each event-0 occurrence, building chains one level at a time with gap-based
+ /// duplicate detection.
int _get_deduplication() const {
- std::vector<TimestampPair> events_timestamp(event_count);
- int max_level = -1;
- int curr_level = -1;
+ int max_level = 0;
+ const size_t list_size = events_list.size();
- for (size_t i = 0; i < events_list.size(); ++i) {
- const auto& evt = events_list[i];
- int event_idx = get_event_idx(evt.event_idx) - 1;
+ for (size_t start = 0; start < list_size; ++start) {
+ // Remaining-events pruning: if remaining events can't beat max_level, stop.
+ if (static_cast<int>(list_size - start) <= max_level) {
+ break;
+ }
- if (event_idx == 0) {
- // Duplicate of event 0: terminate current chain first
- if (events_timestamp[0].has_value()) {
- if (curr_level > max_level) {
- max_level = curr_level;
+ int start_event = get_event_idx(events_list[start].event_idx) - 1;
+ if (start_event != 0) {
+ continue;
+ }
+
+ // Build a chain from this event-0
+ UInt64 first_ts = events_list[start].timestamp;
+ int curr_level = 0;
+ size_t last_advance_idx = start;
+
+ for (size_t i = start + 1; i < list_size; ++i) {
+ int event_idx = get_event_idx(events_list[i].event_idx) - 1;
+
+ // Only search for the next expected level (matches V1's sequential scan)
+ if (event_idx != curr_level + 1) {
+ continue;
+ }
+
+ // Must be from a different row than the last chain event
+ if (_is_same_row(last_advance_idx, i)) {
+ continue;
+ }
+
+ // Window check
+ if (!_within_window(first_ts, events_list[i].timestamp)) {
+ break;
+ }
+
+ // Gap-based dedup check (V1 semantics):
+ // Check if any event at a level below the target (0..event_idx-1)
+ // fired in the gap between the last chain event's row and the
+ // current event's row.
+ size_t gap_start = _next_row_start(last_advance_idx + 1);
+ size_t gap_end = _row_start(i);
+
+ bool dup = false;
+ if (gap_start < gap_end) {
+ for (size_t j = gap_start; j < gap_end; ++j) {
+ int j_level = get_event_idx(events_list[j].event_idx) - 1;
+ if (j_level < event_idx) {
+ dup = true;
+ break;
+ }
}
- _eliminate_chain(curr_level, events_timestamp);
}
- // Start a new chain
- events_timestamp[0] = {evt.timestamp, evt.timestamp, i};
- curr_level = 0;
- } else if (events_timestamp[event_idx].has_value()) {
- // Duplicate event detected: this level was already matched
- if (curr_level > max_level) {
- max_level = curr_level;
+
+ if (dup) {
+ break;
}
- // Eliminate current chain
- _eliminate_chain(curr_level, events_timestamp);
- } else if (events_timestamp[event_idx - 1].has_value() &&
- !_is_same_row(events_timestamp[event_idx - 1].last_list_idx, i)) {
- // Must be from a DIFFERENT row than the predecessor level
- if (_promote_level(events_timestamp, evt.timestamp, i, event_idx, curr_level,
- false)) {
+
+ // Accept this match and advance the chain
+ curr_level = event_idx;
+ last_advance_idx = i;
+
+ if (curr_level + 1 == event_count) {
return event_count;
}
}
- }
- if (curr_level > max_level) {
- return curr_level + 1;
+ max_level = std::max(curr_level + 1, max_level);
}
- return max_level + 1;
+ return max_level;
}
- /// FIXED mode (StarRocks-style semantics): if a matched event appears whose
- /// predecessor level has NOT been matched, the chain is broken (event level jumped).
- /// Note: V2 semantics differ from V1. V1 checks physical row adjacency;
- /// V2 checks event level continuity (unmatched rows don't break the chain).
+ /// FIXED mode (multi-pass, row-based).
+ ///
+ /// Row-by-row semantics similar to V1: after matching c_K at a row, the NEXT
+ /// event-producing row must match c_{K+1} or the chain breaks.
+ ///
+ /// Difference from V1: rows matching no condition ("truly irrelevant") are
+ /// absent from events_list and implicitly skipped. V1 treats them as
+ /// non-matching and breaks the chain. This is the documented 4.1 behavior
+ /// change: irrelevant events no longer break the chain.
+ ///
+ /// Like V1, tries every c1 event as a potential chain starting point and
+ /// returns the maximum level reached across all attempts.
+ ///
+ /// Within each row, events are stored in descending condition-index order:
+ /// [c_max(cont=0), ..., c_min(cont=N)]. The first (non-continuation) event
+ /// carries the row's highest matched condition index. c1 (index 1 after add,
+ /// 0 after -1) is always the last event of its row.
int _get_fixed() const {
- std::vector<TimestampPair> events_timestamp(event_count);
int max_level = -1;
- int curr_level = -1;
- bool first_event = false;
- for (size_t i = 0; i < events_list.size(); ++i) {
- const auto& evt = events_list[i];
- int event_idx = get_event_idx(evt.event_idx) - 1;
+ for (size_t start_i = 0; start_i < events_list.size(); ++start_i) {
+ // Only start from c1 events (condition 0 in 0-based)
+ if (get_event_idx(events_list[start_i].event_idx) != 1) continue;
- if (event_idx == 0) {
- // Save current chain before starting a new one
- if (events_timestamp[0].has_value()) {
- if (curr_level > max_level) {
- max_level = curr_level;
- }
- _eliminate_chain(curr_level, events_timestamp);
- }
- events_timestamp[0] = {evt.timestamp, evt.timestamp, i};
- curr_level = 0;
- first_event = true;
- } else if (first_event && !events_timestamp[event_idx - 1].has_value()) {
- // Event level jumped: predecessor was not matched
- if (curr_level >= 0) {
- if (curr_level > max_level) {
- max_level = curr_level;
- }
- _eliminate_chain(curr_level, events_timestamp);
+ UInt64 chain_start_ts = events_list[start_i].timestamp;
+ int curr_level = 0;
+
+ // c1 is always the last event of its row, so start_i+1 is
+ // the first event of the next row (or end of list).
+ size_t i = start_i + 1;
+
+ while (i < events_list.size() && curr_level + 1 < event_count) {
+ // Skip continuation events to reach the first event of the next row.
+ i = _next_row_start(i);
+ if (i >= events_list.size()) break;
+
+ // Window check against the row's timestamp.
+ if (!_within_window(chain_start_ts, events_list[i].timestamp)) break;
+
+ int expected = curr_level + 1;
+ int highest = get_event_idx(events_list[i].event_idx) - 1;
+
+ if (highest < expected) {
+ // Row's highest matched condition is below expected → break.
+ break;
}
- } else if (events_timestamp[event_idx - 1].has_value() &&
- !_is_same_row(events_timestamp[event_idx - 1].last_list_idx, i)) {
- // Must be from a DIFFERENT row than the predecessor level
- if (_promote_level(events_timestamp, evt.timestamp, i, event_idx, curr_level,
- false)) {
- return event_count;
+
+ if (!_row_contains_level(i, expected)) {
+ // Row matches higher conditions but not the expected one
+ // (level jump) → break.
+ break;
}
+
+ curr_level = expected;
+
+ // Advance past this row to the start of the next one.
+ i = _next_row_start(i + 1);
+ }
+
+ if (curr_level > max_level) {
+ max_level = curr_level;
+ if (max_level + 1 == event_count) return event_count;
}
- }
- if (curr_level > max_level) {
- return curr_level + 1;
+ // Prune: remaining events cannot beat current best.
+ if (static_cast<int>(events_list.size() - start_i) <= max_level) break;
Review Comment:
The pruning check casts (events_list.size() - start_i) from size_t to int. If events_list is large, this can overflow and make the pruning logic behave incorrectly. Use size_t arithmetic/comparison (or clamp) to keep the pruning safe for large inputs.
```suggestion
size_t remaining_events = events_list.size() - start_i;
if (max_level >= 0 && remaining_events <= static_cast<size_t>(max_level)) break;
```
##########
regression-test/suites/nereids_p0/aggregate/window_funnel_v2.groovy:
##########
@@ -488,5 +488,417 @@ suite("window_funnel_v2") {
from windowfunnel_v2_test t;
"""
+ // ==================== DEDUPLICATION mode: same-row multi-event bug fix ====================
+ // Regression test for the bug where deduplication mode incorrectly breaks the chain
+ // when a single row matches multiple conditions including E0.
+ // V1 returns 3, V2 was returning 2 before the fix.
sql """ DROP TABLE IF EXISTS windowfunnel_v2_test """
+ sql """
+ CREATE TABLE IF NOT EXISTS windowfunnel_v2_test (
+ xwho varchar(50) NULL COMMENT 'xwho',
+ xwhen datetimev2(3) COMMENT 'xwhen',
+ xwhat int NULL COMMENT 'xwhat'
+ )
+ DUPLICATE KEY(xwho)
+ DISTRIBUTED BY HASH(xwho) BUCKETS 3
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+ // Case 1: Rows matching both E0+E1 or E0+E2 simultaneously.
+ // Row 0 (t=10:00): xwhat=1 → matches E0 (xwhat=1) and E1 (xwhat!=2)
+ // Row 1 (t=11:00): xwhat=1 → matches E0 (xwhat=1) and E1 (xwhat!=2)
+ // Row 2 (t=12:00): xwhat=3 → matches E0 (xwhat=1 false, but see below), E1 (xwhat!=2) and E2 (xwhat=3)
Review Comment:
This comment says row 2 (xwhat=3) matches E0, but with the listed conditions (xwhat=1, xwhat!=2, xwhat=3) E0 is false on that row. Please update the test description to match the actual predicates/data so future readers aren’t misled.
```suggestion
// Case 1: Rows matching both E0+E1 or E1+E2 simultaneously.
// Row 0 (t=10:00): xwhat=1 → matches E0 (xwhat=1) and E1 (xwhat!=2)
// Row 1 (t=11:00): xwhat=1 → matches E0 (xwhat=1) and E1 (xwhat!=2)
// Row 2 (t=12:00): xwhat=3 → matches E1 (xwhat!=2) and E2 (xwhat=3), while E0 (xwhat=1) is false
```
##########
be/src/exprs/aggregate/aggregate_function_window_funnel_v2.h:
##########
@@ -393,123 +393,209 @@ struct WindowFunnelStateV2 {
if (matched) {
events_timestamp[event_idx] = {events_timestamp[event_idx - 1].first_ts,
evt.timestamp, i};
- if (event_idx > curr_level) {
- curr_level = event_idx;
- }
+ curr_level = std::max(event_idx, curr_level);
if (event_idx + 1 == event_count) {
return event_count;
}
}
}
}
- if (curr_level + 1 > max_level) {
- max_level = curr_level + 1;
- }
+ max_level = std::max(curr_level + 1, max_level);
}
return max_level;
}
- /// DEDUPLICATION mode: if a previously matched event level appears again,
- /// the current chain is terminated and max_level is updated.
- /// This preserves V1 semantics where duplicate events break the chain.
+ /// DEDUPLICATION mode (multi-pass, matching V1 semantics):
+ ///
+ /// V1 tries each E0-matching row as a chain starting point and searches for
+ /// conditions one at a time. Before accepting a match for condition K, V1 checks
+ /// the "gap" (rows between the previous match and the current match) for any
+ /// re-occurrence of already-matched conditions 0..K-1. If found, the chain breaks.
+ ///
+ /// A single-pass approach cannot correctly implement this because V2's greedy
+ /// multi-condition-per-row advancement creates premature higher-level matches
+ /// that then conflict with later events. The multi-pass approach iterates over
+ /// each event-0 occurrence, building chains one level at a time with gap-based
+ /// duplicate detection.
int _get_deduplication() const {
- std::vector<TimestampPair> events_timestamp(event_count);
- int max_level = -1;
- int curr_level = -1;
+ int max_level = 0;
+ const size_t list_size = events_list.size();
- for (size_t i = 0; i < events_list.size(); ++i) {
- const auto& evt = events_list[i];
- int event_idx = get_event_idx(evt.event_idx) - 1;
+ for (size_t start = 0; start < list_size; ++start) {
+ // Remaining-events pruning: if remaining events can't beat max_level, stop.
+ if (static_cast<int>(list_size - start) <= max_level) {
Review Comment:
The pruning condition casts (list_size - start) from size_t to int; for large event lists this can overflow and cause incorrect early termination (wrong result) or missed pruning. Prefer doing the comparison in size_t (e.g., compare to static_cast<size_t>(max_level)) or otherwise clamp safely.
```suggestion
if ((list_size - start) <= static_cast<size_t>(max_level)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](function) fix wrong result of window funnel v2 + deduplication/fixed mode #62043 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62126:
URL: https://github.com/apache/doris/pull/62126#issuecomment-4188610285
# BE Regression && UT Coverage Report
Increment line coverage `95.40% (83/87)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62126_5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62126_5b8e7779c4c4f8bcb8111eb222a692aab8a93d5e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.31% (26112/36617) |
| Line Coverage | 54.18% (275823/509127) |
| Region Coverage | 51.54% (229342/444944) |
| Branch Coverage | 52.95% (99228/187386) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org