You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/10/06 12:03:41 UTC
[PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
SEPURI-SAI-KRISHNA opened a new pull request, #29410:
URL: https://github.com/apache/flink/pull/29410
## What is the purpose of the change
A non-time `OVER` window compares the previous and current accumulator with `RowData#equals` to
decide whether it can skip emitting updates. That comparison is wrong in two ways.
On the heap backend it throws for any aggregate whose accumulator holds a RAW field:
```
java.lang.UnsupportedOperationException: Unmaterialized BinaryRawValueData cannot be compared.
at org.apache.flink.table.data.binary.BinaryRawValueData.equals(BinaryRawValueData.java:85)
at java.base/java.util.Arrays.deepEquals(Arrays.java:4625)
at org.apache.flink.table.data.GenericRowData.equals(GenericRowData.java:229)
at ...NonTimeRangeUnboundedPrecedingFunction.processRemainingElements(...:290)
```
On RocksDB it does not throw but never matches, because the stored accumulator is deserialized
as `BinaryRowData` while the current one is a `GenericRowData`. The early-out never fires, so
every following row is recomputed and emits an `UPDATE_BEFORE`/`UPDATE_AFTER` pair carrying
identical values.
A generated `RecordEqualiser` over the accumulator types fixes both: it materializes RAW fields
before comparing them, and it reads fields positionally so the two row classes may differ.
## Brief change log
- `StreamExecOverAggregate` generates a `RecordEqualiser` for the flattened accumulator types and
passes it to both non-time functions.
- `AbstractNonTimeUnboundedPrecedingOver` and `NonTimeRangeUnboundedPrecedingFunction` use it for
the early-out comparison.
- `NonTimeOverAggregateITCase` covers `LAG`, `ARRAY_AGG`, `COLLECT` and `PERCENTILE` on a window
ordered by a non-time attribute, under both state backends.
The two remaining comparisons in `NonTimeRowsUnboundedPrecedingFunction` compare an aggregate
value against an accumulator, which FLINK-40883 is correcting. They are left alone here; the
equaliser is threaded through its constructor so it is available once that lands. The two changes
touch different lines and can land in either order.
`COLLECT` and `PERCENTILE` values are not asserted, only that the queries run. They return the
current row rather than the running window, which is FLINK-40737.
## Verifying this change
Red/green on the new ITCase, which runs 8 cases (4 queries x 2 state backends). With the two
comparison changes reverted, 4 of the 8 fail: every heap case, each with the exception above. The
4 RocksDB cases pass, emitting the redundant pairs described. All 8 pass with the change.
Existing coverage: `NonTimeRangeUnboundedPrecedingFunctionTest` and
`NonTimeRowsUnboundedPrecedingFunctionTest` 24/24; the `OverAggregate`/`OverWindow` planner, ITCase,
harness and restore suites 386 tests, 0 failures, including `OverWindowRestoreTest` and
`OverAggregateRestoreTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (yes, the early-out comparison in a
non-time OVER window is now a generated equaliser instead of `RowData#equals`)
- Anything that affects deployment or recovery: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4244579828
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -422,6 +436,36 @@ RowData setAccumulatorAndGetValue(RowData accumulator) throws Exception {
return aggFuncs.getValue();
}
+ /**
+ * Reads the accumulator stored for a sort key, as a copy the caller owns.
+ *
+ * <p>An accumulator can hold mutable content, either a data view or a field such as a bitmap,
+ * an array or a {@code byte[]}. The aggregate functions change that content in place, and the
+ * heap state backend hands back the very object it stores. Without a copy, accumulating into
+ * one sort key's accumulator would also change the one state holds for another.
+ *
+ * @param accKey the sort key to read the accumulator of
+ * @return a private copy of the stored accumulator, or null if there is none
+ */
+ RowData getAccFromState(RowData accKey) throws Exception {
+ final RowData acc = accMapState.get(accKey);
+ return acc == null ? null : accSerializer.copy(acc);
Review Comment:
I tried it: copy only inside setAccumulatorOfPrevRow and setAccumulatorOfPrevId, plain get elsewhere. Green, 26/26 in NonTimeOverAggregateITCase and 54/54 in the harness tests.
That only says the suite does not catch it. Those other reads are safe because each one is handed to setAccumulators and the handler is pointed at another accumulator before anything accumulates into it. Nothing enforces that ordering, and the earlier version of this patch already shipped a hole of exactly that kind by gating on the accumulator type and missing BITMAP. So I would rather keep the copy at the state boundary. Happy to narrow it if you prefer.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -422,6 +436,36 @@ RowData setAccumulatorAndGetValue(RowData accumulator) throws Exception {
return aggFuncs.getValue();
}
+ /**
+ * Reads the accumulator stored for a sort key, as a copy the caller owns.
+ *
+ * <p>An accumulator can hold mutable content, either a data view or a field such as a bitmap,
+ * an array or a {@code byte[]}. The aggregate functions change that content in place, and the
+ * heap state backend hands back the very object it stores. Without a copy, accumulating into
+ * one sort key's accumulator would also change the one state holds for another.
+ *
+ * @param accKey the sort key to read the accumulator of
+ * @return a private copy of the stored accumulator, or null if there is none
+ */
+ RowData getAccFromState(RowData accKey) throws Exception {
+ final RowData acc = accMapState.get(accKey);
+ return acc == null ? null : accSerializer.copy(acc);
+ }
+
+ /**
+ * Stores the accumulator of a sort key, as a copy state owns.
+ *
+ * <p>The aggregate functions keep changing their accumulator after it has been stored, and the
+ * reset at the end of {@link #processElement} clears its data views. The heap state backend
+ * keeps the object it was given, so the copy is what keeps the stored accumulator intact.
+ *
+ * @param accKey the sort key to store the accumulator of
+ * @param acc the accumulator to store
+ */
+ void putAccInState(RowData accKey, RowData acc) throws Exception {
+ accMapState.put(accKey, accSerializer.copy(acc));
Review Comment:
The write copy has to stay. Dropping it on its own fails 8 of 26 in NonTimeOverAggregateITCase, all on the HEAP run: LAG, ARRAY_AGG, BITMAP_BUILD_CARDINALITY, COLLECT and COUNT(DISTINCT). The harness tests stay 54/54, so the ITCase is the only thing that catches it.
An allowlist on the aggregate would not be safe. COUNT(DISTINCT v) is one of those 8 and its accumulator carries a MapView, so seeing COUNT tells you nothing about the fields. Checking every field serializer for isImmutableType would be sound, MapViewSerializer and BitmapSerializer both return false, but I do not think it earns the extra branch here.
Every record already reads the whole sorted list back through sortedListState.value() and scans it linearly in findIndexOfSortKey. An append copies two accumulators, and an insert in between copies two more per following row, each paired with a state get or put that costs more than the copy itself. If we want to cut work on this path, the state round trips are where it is, not the copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217555028
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/NonTimeRowsUnboundedPrecedingFunction.java:
##########
@@ -99,6 +100,7 @@ public NonTimeRowsUnboundedPrecedingFunction(
genAggsHandler,
genRecordEqualiser,
genSortKeyEqualiser,
+ genAccEqualiser,
Review Comment:
Agreed, it cannot land in either order as it stands. Happy to rebase on 40883 once it is in and move its comparisons to accEqualiser, or to do that here if this lands second. Whichever you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217538776
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -195,6 +199,10 @@ public void open(OpenContext openContext) throws Exception {
sortKeyEqualiser =
generatedSortKeyEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ // Initialize accumulator equaliser
+ accEqualiser =
+ generatedAccEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
Review Comment:
Done, and this is better than what I had. Every get and put of accMapState now goes through getAccFromState and putAccInState, which copy with the accumulator serializer every time, for every accumulator. The remove and the TTL cleanupState still use the state directly, neither of them hands an accumulator out.
My version gated the copy on the accumulator holding a RAW field, which was too narrow. BITMAP is
its own LogicalTypeRoot, so BITMAP_BUILD_CARDINALITY_AGG was never copied, and on the same data
heap returned {10=1, 20=3, 25=4, 30=2} against {10=1, 20=2, 25=3, 30=4} on RocksDB. Added
testBitmapBuildCardinalityInsertsInBetween.
The copy on put is also what the reset at the end of processElement needed, so I dropped the
conditional copy I had on FLINK-40737.
Red/green on this branch: 4 of the 12 runs fail without the copy, all of them on heap, and all 12
pass with it. The four are the tests named above.
One trade off worth naming: the copy is unconditional, so it also runs on paths that only read an
accumulator, and on RocksDB where the backend already hands back its own copy. I kept it uniform
because narrowing it is what produced the bug I just fixed. Happy to restrict it if you would
rather not pay that on the per record path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217538776
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -195,6 +199,10 @@ public void open(OpenContext openContext) throws Exception {
sortKeyEqualiser =
generatedSortKeyEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ // Initialize accumulator equaliser
+ accEqualiser =
+ generatedAccEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
Review Comment:
Done, and this is better than what I had. Every get and put of accMapState now goes through getAccFromState and putAccInState, which copy with the accumulator serializer every time, for every accumulator. The remove and the TTL cleanupState still use the state directly, neither of them hands an accumulator out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #29410:
URL: https://github.com/apache/flink/pull/29410#issuecomment-6065146372
@SEPURI-SAI-KRISHNA I think you can rebase and take the next actions here, also for #29412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4205091194
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -195,6 +199,10 @@ public void open(OpenContext openContext) throws Exception {
sortKeyEqualiser =
generatedSortKeyEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ // Initialize accumulator equaliser
+ accEqualiser =
+ generatedAccEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
Review Comment:
I think the copy belongs here: copy on `accMapState` get and put, for every accumulator. With that, LAG, ARRAY_AGG, BITMAP_BUILD_AGG and array or byte[] UDAFs are correct on heap.
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecOverAggregate.java:
##########
@@ -407,6 +407,12 @@ private KeyedProcessFunction<RowData, RowData, RowData> createUnboundedOverProce
new EqualiserCodeGenerator(sortKeyRowType, ctx.classLoader())
.generateRecordEqualiser("FirstMatchingSortKeyEqualiser");
+ // RowData#equals throws on a RAW accumulator field and never matches a
+ // BinaryRowData read back from state, so the comparison has to be generated.
+ final GeneratedRecordEqualiser generatedAccEqualiser =
Review Comment:
State backed views compare equal here, so without FLINK-40737 heap COLLECT and COUNT(DISTINCT) go from an exception to wrong results. I'd like this to land together with that PR.
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/NonTimeOverAggregateITCase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.stream.sql;
+
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.planner.runtime.utils.StreamingWithStateTestBase;
+import org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.entry;
+
+/** Tests for OVER windows ordered by a non-time attribute. */
+@ExtendWith(ParameterizedTestExtension.class)
+public class NonTimeOverAggregateITCase extends StreamingWithStateTestBase {
+
+ public NonTimeOverAggregateITCase(StateBackendMode state) {
+ super(state);
+ }
+
+ @Parameters(name = "backend = {0}")
+ public static Collection<Object[]> parameters() {
+ return Arrays.asList(
+ new Object[][] {
+ {StreamingWithStateTestBase.HEAP_BACKEND()},
+ {StreamingWithStateTestBase.ROCKSDB_BACKEND()}
+ });
+ }
+
+ @BeforeEach
+ public void before() {
+ super.before();
+ String dataId =
+ TestValuesTableFactory.registerData(
+ Arrays.asList(
+ Row.of("a", 30, "p"), Row.of("a", 20, "q"), Row.of("a", 10, "r")));
Review Comment:
These rows only insert at the front and UPDATE_BEFORE values are skipped below, so this doesn't show. Can you add a row that lands in between and check the retractions?
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/NonTimeRangeUnboundedPrecedingFunction.java:
##########
@@ -287,7 +289,7 @@ void processRemainingElements(
// Get previous accumulator
RowData prevAcc = accMapState.get(curSortKey);
- if (prevAcc.equals(currAcc)) {
+ if (accEqualiser.equals(prevAcc, currAcc)) {
Review Comment:
On heap, `accMapState.get` returns the stored accumulator and LAG and ARRAY_AGG append to it in place. Rows 30, 20, 25 give ARRAY_AGG `[q, p, s]` for 25.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/NonTimeRowsUnboundedPrecedingFunction.java:
##########
@@ -99,6 +100,7 @@ public NonTimeRowsUnboundedPrecedingFunction(
genAggsHandler,
genRecordEqualiser,
genSortKeyEqualiser,
+ genAccEqualiser,
Review Comment:
This can't land in either order with FLINK-40883: its tests use the old constructor and its LAST_VALUE tests fail on `TestAccEqualiser`. The second one to land also moves its comparisons to `accEqualiser`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "bvarghese1 (via GitHub)" <gi...@apache.org>.
bvarghese1 commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4235478367
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -422,6 +436,36 @@ RowData setAccumulatorAndGetValue(RowData accumulator) throws Exception {
return aggFuncs.getValue();
}
+ /**
+ * Reads the accumulator stored for a sort key, as a copy the caller owns.
+ *
+ * <p>An accumulator can hold mutable content, either a data view or a field such as a bitmap,
+ * an array or a {@code byte[]}. The aggregate functions change that content in place, and the
+ * heap state backend hands back the very object it stores. Without a copy, accumulating into
+ * one sort key's accumulator would also change the one state holds for another.
+ *
+ * @param accKey the sort key to read the accumulator of
+ * @return a private copy of the stored accumulator, or null if there is none
+ */
+ RowData getAccFromState(RowData accKey) throws Exception {
+ final RowData acc = accMapState.get(accKey);
+ return acc == null ? null : accSerializer.copy(acc);
Review Comment:
This affects the hot path. Should we copy only on the reads that need it i.e. from `setAccumulatorOfPrevRow/setAccumulatorOfPrevId` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29410:
URL: https://github.com/apache/flink/pull/29410#issuecomment-6016041340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f19b1091f70736f54572b72284b4a0312d83d1ef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f19b1091f70736f54572b72284b4a0312d83d1ef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f19b1091f70736f54572b72284b4a0312d83d1ef UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217551767
##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/NonTimeOverAggregateITCase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.runtime.stream.sql;
+
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import org.apache.flink.table.planner.runtime.utils.StreamingWithStateTestBase;
+import org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
+import org.apache.flink.testutils.junit.extensions.parameterized.Parameters;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.CloseableIterator;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.entry;
+
+/** Tests for OVER windows ordered by a non-time attribute. */
+@ExtendWith(ParameterizedTestExtension.class)
+public class NonTimeOverAggregateITCase extends StreamingWithStateTestBase {
+
+ public NonTimeOverAggregateITCase(StateBackendMode state) {
+ super(state);
+ }
+
+ @Parameters(name = "backend = {0}")
+ public static Collection<Object[]> parameters() {
+ return Arrays.asList(
+ new Object[][] {
+ {StreamingWithStateTestBase.HEAP_BACKEND()},
+ {StreamingWithStateTestBase.ROCKSDB_BACKEND()}
+ });
+ }
+
+ @BeforeEach
+ public void before() {
+ super.before();
+ String dataId =
+ TestValuesTableFactory.registerData(
+ Arrays.asList(
+ Row.of("a", 30, "p"), Row.of("a", 20, "q"), Row.of("a", 10, "r")));
Review Comment:
Added. testRetractionsOfInsertInBetween asserts the raw changelog for that insert, so the UPDATE_BEFORE values are checked rather than collapsed away by the helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4230538930
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -195,6 +199,10 @@ public void open(OpenContext openContext) throws Exception {
sortKeyEqualiser =
generatedSortKeyEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ // Initialize accumulator equaliser
+ accEqualiser =
+ generatedAccEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
Review Comment:
Suits me, left it unconditional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217533925
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/NonTimeRangeUnboundedPrecedingFunction.java:
##########
@@ -287,7 +289,7 @@ void processRemainingElements(
// Get previous accumulator
RowData prevAcc = accMapState.get(curSortKey);
- if (prevAcc.equals(currAcc)) {
+ if (accEqualiser.equals(prevAcc, currAcc)) {
Review Comment:
Confirmed. src_repeated inserts 25 between 20 and 30, and on heap that gives ARRAY_AGG [r, p, q] for 20, [r, p, q, p, r] for 25 and [r, p, q, p, r, p] for 30, against [r, q], [r, q, r] and [r, q, r, p] on RocksDB. LAG gives 20=p where RocksDB gives 20=r. Added testArrayAggInsertsInBetween and testLagInsertsInBetween.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "bvarghese1 (via GitHub)" <gi...@apache.org>.
bvarghese1 commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4235496199
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -422,6 +436,36 @@ RowData setAccumulatorAndGetValue(RowData accumulator) throws Exception {
return aggFuncs.getValue();
}
+ /**
+ * Reads the accumulator stored for a sort key, as a copy the caller owns.
+ *
+ * <p>An accumulator can hold mutable content, either a data view or a field such as a bitmap,
+ * an array or a {@code byte[]}. The aggregate functions change that content in place, and the
+ * heap state backend hands back the very object it stores. Without a copy, accumulating into
+ * one sort key's accumulator would also change the one state holds for another.
+ *
+ * @param accKey the sort key to read the accumulator of
+ * @return a private copy of the stored accumulator, or null if there is none
+ */
+ RowData getAccFromState(RowData accKey) throws Exception {
+ final RowData acc = accMapState.get(accKey);
+ return acc == null ? null : accSerializer.copy(acc);
+ }
+
+ /**
+ * Stores the accumulator of a sort key, as a copy state owns.
+ *
+ * <p>The aggregate functions keep changing their accumulator after it has been stored, and the
+ * reset at the end of {@link #processElement} clears its data views. The heap state backend
+ * keeps the object it was given, so the copy is what keeps the stored accumulator intact.
+ *
+ * @param accKey the sort key to store the accumulator of
+ * @param acc the accumulator to store
+ */
+ void putAccInState(RowData accKey, RowData acc) throws Exception {
+ accMapState.put(accKey, accSerializer.copy(acc));
Review Comment:
This is also on the hot path. Can we skip the copy when it doesnt matter? If every accumulator field is immutable, the generated handler never changes the stored row in place. This can be done once in the `open()` and we can skip for the most common cases ie. `COUNT/SUM/MIN/MAX/AVG/ROW_NUMBER/RANK`.
If its indeed cheap like @MartijnVisser suggested I would avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4228880356
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/over/AbstractNonTimeUnboundedPrecedingOver.java:
##########
@@ -195,6 +199,10 @@ public void open(OpenContext openContext) throws Exception {
sortKeyEqualiser =
generatedSortKeyEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
+ // Initialize accumulator equaliser
+ accEqualiser =
+ generatedAccEqualiser.newInstance(getRuntimeContext().getUserCodeClassLoader());
Review Comment:
Keep it unconditional. Narrowing it is how BITMAP slipped through, and the extra copy is cheap next to what this operator already does per insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40735][table] Compare non-time OVER accumulators with a generated equaliser [flink]
Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on code in PR #29410:
URL: https://github.com/apache/flink/pull/29410#discussion_r4217548722
##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/stream/StreamExecOverAggregate.java:
##########
@@ -407,6 +407,12 @@ private KeyedProcessFunction<RowData, RowData, RowData> createUnboundedOverProce
new EqualiserCodeGenerator(sortKeyRowType, ctx.classLoader())
.generateRecordEqualiser("FirstMatchingSortKeyEqualiser");
+ // RowData#equals throws on a RAW accumulator field and never matches a
+ // BinaryRowData read back from state, so the comparison has to be generated.
+ final GeneratedRecordEqualiser generatedAccEqualiser =
Review Comment:
Agreed, 29412 is stacked on this branch. I checked the split holds: with the copy but without the 40737 change, only the six data view tests fail, and those fail on both backends. So this PR fixes ARRAY_AGG, LAG and the BITMAP aggregates on heap, and 40737 fixes COLLECT, PERCENTILE and the distinct aggregates on both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org