You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ldadima (via GitHub)" <gi...@apache.org> on 2026/03/13 04:02:35 UTC
[PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
ldadima opened a new pull request, #27764:
URL: https://github.com/apache/flink/pull/27764
## What is the purpose of the change
To fix [FLINK-39015](https://issues.apache.org/jira/browse/FLINK-39015)
## Verifying this change
Run JoinITCase#testThreeWayMultiJoinWithoutPk
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? ( no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2966252255
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/JoinITCase.scala:
##########
@@ -640,6 +640,55 @@ class JoinITCase(miniBatch: MiniBatchMode, state: StateBackendMode, enableAsyncS
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+ @TestTemplate
+ def testThreeWayMultiJoinWithoutPk(): Unit = {
Review Comment:
Ideally would be great to follow new store restore test approach
would be great if you can do that instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2972721804
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/JoinITCase.scala:
##########
@@ -640,6 +640,55 @@ class JoinITCase(miniBatch: MiniBatchMode, state: StateBackendMode, enableAsyncS
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+ @TestTemplate
+ def testThreeWayMultiJoinWithoutPk(): Unit = {
Review Comment:
This test, as in the [previous PR](https://github.com/apache/flink/pull/27508) with a similar problem, is only temporary. It was decided to expand this [PR](https://github.com/apache/flink/pull/27415) with the necessary tests.
Is it ok, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4152505934
> A general performance comparison considering some aspects would be great to share with the community. Wdyt?
Oh, sorry. I didn't understand you earlier. Indeed, it would be good to make smth like that
But we need to understand what we want to compare. There is Q23 (different data distributions) and flink-benchmarks
Maybe it's also worth creating a Jira ticket for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2966257851
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/state/MultiJoinStateViews.java:
##########
@@ -313,16 +321,19 @@ public Iterator<RowData> iterator() {
private static final class InputSideHasNoUniqueKey implements MultiJoinStateView {
private final MapState<RowData, Integer> recordState;
private RowDataSerializer joinKeySerializer; // Null if joinKeyType is null
+ private RowDataSerializer stateKeySerializer; // Null if joinKeyType is null
private int joinKeyFieldCount; // 0 if joinKeyType is null
private final int recordFieldCount;
@Nullable private final RowType joinKeyType; // Store to check for null
+ private final boolean prohibitReuseRowData;
private InputSideHasNoUniqueKey(
RuntimeContext ctx,
final String stateName,
@Nullable final RowType joinKeyType, // Can be null
final RowType recordType,
- final StateTtlConfig ttlConfig) {
+ final StateTtlConfig ttlConfig,
+ boolean prohibitReuseRowData) {
Review Comment:
```suggestion
final boolean prohibitReuseRowData) {
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2972721804
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/JoinITCase.scala:
##########
@@ -640,6 +640,55 @@ class JoinITCase(miniBatch: MiniBatchMode, state: StateBackendMode, enableAsyncS
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+ @TestTemplate
+ def testThreeWayMultiJoinWithoutPk(): Unit = {
Review Comment:
@snuyanzin Thanks for review
This test, as in the [previous PR](https://github.com/apache/flink/pull/27508) with a similar problem, is only temporary. It was decided to expand this [PR](https://github.com/apache/flink/pull/27415) with the necessary tests.
Is it ok, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2996969662
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/state/MultiJoinStateViews.java:
##########
@@ -68,7 +68,8 @@ public static MultiJoinStateView create(
RowType
joinKeyType, /* joinKeyType is null for inputId = 0, see {@link InputSideHasUniqueKey}*/
RowType recordType,
- long retentionTime) {
+ long retentionTime,
+ boolean prohibitReuseRowData) {
Review Comment:
nit: Rename `prohibitReuseRowData` to `requiresDeepCopy` for consistency with `AttributeBasedJoinKeyExtractor` from #27508. Apply across both inner classes too.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/StreamingMultiJoinOperator.java:
##########
@@ -835,7 +835,8 @@ private void initializeStateHandlers() {
inputSpecs.get(i),
joinKeyType,
inputTypes.get(i),
- stateRetentionTime[i]);
+ stateRetentionTime[i],
+ prohibitReuseRow);
Review Comment:
nit: Rename to `requiresDeepCopy` to match #27508 naming.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/state/MultiJoinStateViews.java:
##########
@@ -231,7 +239,7 @@ private RowData getStateKey(RowData joinKey, RowData uniqueKey) {
GenericRowData compositeKey = new GenericRowData(2);
compositeKey.setField(0, joinKey);
compositeKey.setField(1, uniqueKey);
- return compositeKey;
+ return stateKeySerializer.toBinaryRow(compositeKey, prohibitReuseRowData);
Review Comment:
Two things:
1. Add a comment explaining why, e.g.: `// Serialize to BinaryRowData to prevent object reuse corruption on heap state backend`
2. Skip conversion entirely for RocksDB - it serializes state keys through TypeSerializer anyway. Same pattern as `AttributeBasedJoinKeyExtractor`:
```java
if (!requiresDeepCopy) {
return compositeKey;
}
return stateKeySerializer.toBinaryRow(compositeKey, true);
```
Apply to `InputSideHasNoUniqueKey.getStateKey()` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2965734024
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/stream/state/MultiJoinStateViews.java:
##########
@@ -191,7 +196,8 @@ private InputSideHasUniqueKey(
final RowType recordType,
final InternalTypeInfo<RowData> uniqueKeyType,
final KeySelector<RowData, RowData> uniqueKeySelector,
- final StateTtlConfig ttlConfig) {
+ final StateTtlConfig ttlConfig,
+ boolean prohibitReuseRowData) {
Review Comment:
is there a reason we have `final` for all args except the last one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4140437446
Thanks for review @gustavodemorais .
Code changed according your comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4141684914
Thanks @gustavodemorais , I will use this opportunity (Slack) if necessary.
I used this benchmark (https://github.com/nexmark/nexmark)
There is a prepared set of SQL queries. I took part of the set (significant queries with joins) and compared the results with multi-join optimization turned off and on. Also in the nexmark benchmarks, there is Q23 - a query with multiple joins, where the results of multi-join optimization can be seen.
Besides comparing performance, it is also worth noting that regular join supports Window and mini-batch improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4052470382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9f402c3ac96801919bfe82ebaaed55398a7f957",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9f402c3ac96801919bfe82ebaaed55398a7f957",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9f402c3ac96801919bfe82ebaaed55398a7f957 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4142013295
I think a general performance comparison is beyond the scope of the task described. In the Jira ticket I created, I wanted to highlight the issue of multi-join with two inputs. I don't think it's necessary to consider the case where the multi-join rule generate multi-join with more than two inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #27764:
URL: https://github.com/apache/flink/pull/27764#discussion_r2999754626
##########
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/JoinITCase.scala:
##########
@@ -640,6 +640,55 @@ class JoinITCase(miniBatch: MiniBatchMode, state: StateBackendMode, enableAsyncS
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+ @TestTemplate
+ def testThreeWayMultiJoinWithoutPk(): Unit = {
Review Comment:
Yeah, let's add also this one to the new suite. I'll take a look at the new suite next
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "ldadima (via GitHub)" <gi...@apache.org>.
ldadima commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4054051139
Hi @gustavodemorais
Can you reciew, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4142604913
> I think a general performance comparison is beyond the scope of the task described.
I wasn't referring to this PR or another jira you created. I'm talking about another possible ticket. A general performance comparison considering some aspects would be great to share with the community. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4141882079
I know Q23 and it'd be nice to have some statistic on it for different type of data coming in. In the example of Q23, how does the performance progresses if "B.bidder" has only one record in each table, or 10 records, or 1000 records etc.
The same would be for a larger join, like 10 joins. Do you think https://github.com/apache/flink-benchmarks would be a good way to have this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #27764:
URL: https://github.com/apache/flink/pull/27764#issuecomment-4161157400
Thanks for the PR @ldadima
thanks for the review @gustavodemorais
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39241][table] Fix MultiJoinStateViews by changing GenericRowData to BinaryRowData [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #27764:
URL: https://github.com/apache/flink/pull/27764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org