You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "naivedogger (via GitHub)" <gi...@apache.org> on 2026/07/17 04:28:42 UTC
[PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
naivedogger opened a new pull request, #3675:
URL: https://github.com/apache/fluss/pull/3675
### Purpose
Linked issue: close #3670
Implement `SupportsLookupCustomShuffle` (FLIP-462) in the Flink 2.2 connector so a
lookup-join probe stream is repartitioned by the Fluss bucket key. Rows that map to the
same Fluss bucket are co-located on the same lookup subtask, which improves lookup cache
locality and lowers RPC fan-out. The feature is opt-in through Flink's existing
`LOOKUP('shuffle'='true')` join hint; behaviour is unchanged when the hint is absent.
### Brief change log
- **fluss-flink-2.2**: add `FlinkLookupShuffleTableSource` (extends `FlinkTableSource`,
implements `SupportsLookupCustomShuffle`). `getPartitioner()` returns a bucket-based
partitioner for full primary-key lookups; `copy()` preserves the subtype so the ability
survives planner copies.
- **fluss-flink-2.2**: add `FlussLookupInputPartitioner`, which mirrors client-side
bucketing (bucket-key encoding + `BucketingFunction`, selected by the table's data-lake
format) and routes `bucketId % numPartitions`, matching how `PrimaryKeyLookuper` buckets.
- **Cross-version isolation**: add a same-FQN `LookupShuffleSourceAdapter` in both
`fluss-flink-common` (no-op) and `fluss-flink-2.2` (wraps the source). The 2.x class
shadows the common one at package time, keeping `fluss-flink-common` compilable against
Flink 1.20 (`SupportsLookupCustomShuffle` does not exist before Flink 2.0). This follows
the existing `*Adapter` pattern in the module.
- **fluss-flink-common `FlinkTableSource`**: add a copy constructor as the single source of
truth for state cloning (`copy()` now delegates to it); stash the `LookupNormalizer`
computed in `getLookupRuntimeProvider()` for later use by `getPartitioner()`; expose a few
`protected` accessors for the 2.x variant.
- **fluss-flink-common `FlinkTableFactory`**: route the created source through
`LookupShuffleSourceAdapter.maybeWithCustomShuffle(...)`.
### Tests
- `FlussLookupInputPartitionerTest` (unit, 4 cases): `partition()` equals `bucket % numPartitions`,
is deterministic, routes same-bucket keys to the same partition, and handles a composite
bucket key — verified against an independent bucket oracle.
- `Flink22LookupShuffleITCase` (IT, 3 cases):
- Non-partitioned and partitioned PK dim tables: inspect the `StreamGraph` and assert the
lookup-join probe edge is partitioned by Flink's `RowDataCustomStreamPartitioner` (i.e.
our partitioner was actually applied) only when the hint is present; plus result parity
with/without the hint (custom shuffle is a pure distribution optimization).
- `testSameBucketKeysAreRoutedToSameSubtask`: end-to-end via `partitionCustom` using the
production partitioner — same bucket → same subtask, `subtask == bucketId % parallelism`,
and keys spread across more than one subtask.
- `mvn clean verify` passes for the affected modules (`fluss-flink-common`, `fluss-flink-2.2`);
spotless + checkstyle clean.
### API and Format
No changes to public Fluss API or storage/wire format. This only adds a Flink 2.x
`DynamicTableSource` ability (`SupportsLookupCustomShuffle`) on the Flink connector side.
### Documentation
No user-facing docs required; the feature is opt-in via the standard Flink
`LOOKUP('shuffle'='true')` hint. Current scope/limitations:
- Enabled for full primary-key lookups only; prefix lookups keep Flink's default distribution.
- Partitioned tables are shuffled by bucket only (no `(partition, bucket)` alignment).
- A temporary table created without an explicit `bucket.num` silently keeps the default
distribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-4999793552
thanks for your work! @naivedogger . I have a minor question about this.
> which improves lookup cache locality
Could you help describe more about this? I haven't seen the similar lookup cache related code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3903005642
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/FlinkTableSource.java:
##########
@@ -532,11 +549,41 @@ public LookupRuntimeProvider getLookupRuntimeProvider(LookupContext context) {
}
}
+ @Nullable
+ private InputDataPartitionerAdapter createLookupInputPartitioner(
+ LookupNormalizer lookupNormalizer) {
+ if (numBuckets == null) {
Review Comment:
Agreed. The eligibility check is now based on whether bucket keys are present. Without bucket keys, the source returns an empty partitioner and Flink uses its default distribution. bucket.num is only read as a local primitive int when creating the bucket partitioner, so no factory or constructor changes are needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3902019478
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/FlinkTableSource.java:
##########
@@ -532,11 +549,41 @@ public LookupRuntimeProvider getLookupRuntimeProvider(LookupContext context) {
}
}
+ @Nullable
+ private InputDataPartitionerAdapter createLookupInputPartitioner(
+ LookupNormalizer lookupNormalizer) {
+ if (numBuckets == null) {
Review Comment:
Could we use the presence of bucket keys, rather than numBuckets == null, to determine whether bucket shuffle should be applied?
At this point, every Fluss table already has a resolved bucket count, and `FlinkConversions#toFlinkTable `always exposes it through `bucket.num`. Bucket keys are what distinguish hash distribution from random/sticky distribution.
Therefore, numBuckets should be modeled as a non-null int and used only to calculate the bucket ID. The bucket-shuffle eligibility check should be based on whether bucket keys are present.
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/lookup/FlussLookupInputPartitioner.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.source.lookup;
+
+import org.apache.fluss.bucketing.BucketingFunction;
+import org.apache.fluss.flink.adapter.SupportsLookupCustomShuffleAdapter.InputDataPartitionerAdapter;
+import org.apache.fluss.flink.row.FlinkAsFlussRow;
+import org.apache.fluss.flink.utils.FlinkConversions;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedKeyEncoder;
+import org.apache.fluss.row.encode.KeyEncoder;
+import org.apache.fluss.utils.MathUtils;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+
+import javax.annotation.Nullable;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+
+/**
+ * Partitions the lookup-join probe stream by Fluss bucket, consistent with the client-side
+ * bucketing used by {@code PrimaryKeyLookuper}/{@code PrefixKeyLookuper} (bucket-key encoding +
+ * {@link BucketingFunction}).
+ *
+ * <p>Partitioned and non-partitioned tables use the same strategy. When there are fewer buckets
+ * than lookup subtasks, each bucket is assigned a subset of subtasks and normalized lookup keys are
+ * hashed within that subset. This avoids leaving lookup subtasks idle for small-bucket tables while
+ * keeping every lookup key on a stable subtask and bounding each bucket's RPC fan-out.
+ */
+public class FlussLookupInputPartitioner implements InputDataPartitionerAdapter {
+
+ private static final long serialVersionUID = 1L;
+
+ private final LookupNormalizer normalizer;
+ // Flink row type of the normalized lookup key row (the expected lookup keys in Fluss key order,
+ // i.e. the full primary key for a primary-key lookup, or the bucket keys + partition keys for a
+ // prefix lookup).
+ private final RowType keyFlinkRowType;
+ private final List<String> bucketKeyNames;
+ @Nullable private final DataLakeFormat lakeFormat;
+ private final int numBuckets;
+
+ private transient KeyEncoder bucketKeyEncoder;
+ private transient KeyEncoder lookupKeyEncoder;
+ private transient BucketingFunction bucketingFunction;
+ private transient FlinkAsFlussRow reuseRow;
+
+ /**
+ * Creates a partitioner consistent with Fluss client-side bucket routing.
+ *
+ * @param normalizer normalizes Flink lookup keys into Fluss lookup-key order
+ * @param keyFlinkRowType row type of the normalized lookup key
+ * @param bucketKeyNames bucket-key field names within the normalized lookup key
+ * @param lakeFormat optional lake format that defines key encoding and bucketing behavior
+ * @param numBuckets positive number of buckets in the Fluss table
+ */
+ public FlussLookupInputPartitioner(
+ LookupNormalizer normalizer,
+ RowType keyFlinkRowType,
+ List<String> bucketKeyNames,
+ @Nullable DataLakeFormat lakeFormat,
+ int numBuckets) {
+ this.normalizer = normalizer;
+ this.keyFlinkRowType = keyFlinkRowType;
+ this.bucketKeyNames = bucketKeyNames;
+ this.lakeFormat = lakeFormat;
+ checkArgument(numBuckets > 0, "numBuckets must be positive, but was %s.", numBuckets);
Review Comment:
add same check as org.apache.fluss.flink.source.lookup.LookupNormalizer#validateAndCreateLookupNormalizer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5600621849
Thanks @naivedogger for the update, LGTM. wait the CI green...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5438822256
Thanks @loserwang1024 for the detailed review! I have pushed a commit addressing your comments. PTAL when convenient!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5491278428
> Hi @loserwang1024, I have one more question about partitioned tables and would appreciate your opinion.
>
> I checked Paimon's current implementation and aligned this PR with it. Paimon's `BucketIdExtractor` derives only the bucket id, and partitioned and non-partitioned tables share the same `BucketShuffleStrategy`. The revised Fluss implementation now behaves the same way:
>
> * when `bucket.num >= lookup parallelism`, use `bucketId % parallelism`;
> * when `bucket.num < lookup parallelism`, assign each bucket a round-robin subtask subset and hash the complete normalized lookup key within that subset. For partitioned tables, that complete key includes the partition key.
>
> This keeps the same lookup key stable and uses all lookup subtasks. The trade-off is that one `(partition, bucket)` tablet may be accessed by multiple subtasks when the bucket count is small. Before this PR, Flink's regular hash shuffle already had this property, so it is not a correctness regression; a more specialized partition/tablet-affinity optimization could be handled separately later.
>
> Do you think keeping partitioned and non-partitioned tables consistent in this PR is reasonable, or do you see a need to preserve `(partition, bucket)` affinity here?
The current assignment makes every downstream subtask reachable, but does not utilize them evenly when the number of subtasks is not divisible by the number of buckets. For example, with 3 buckets and 10 subtasks, the buckets are assigned 4, 3, and 3 candidate subtasks respectively. Assuming similar traffic across buckets, subtasks assigned to the first bucket receive only one quarter of that bucket’s traffic, while the others receive one third.
Consider using a weighted logical-slot assignment similar to WeightedBucketIdAssignment. For example, 3 buckets and 10 subtasks can be represented by 30 logical slots, with 10 slots per bucket and 3 slots per physical subtask. A deterministic hash of the normalized lookup key can then select a logical slot within its bucket. This would preserve deterministic routing and bounded bucket fan-out while distributing the expected load evenly across all downstream subtasks.
```mermaid
flowchart TB
A["Assumption: 3 buckets have equal traffic<br/>10 downstream sink subtasks"]
subgraph CURRENT["Current algorithm — disjoint candidate sets"]
direction TB
C0["Bucket 0<br/>1/3 traffic"] --> C0S["Sinks 0, 3, 6, 9<br/>each receives 1/12 = 8.3%"]
C1["Bucket 1<br/>1/3 traffic"] --> C1S["Sinks 1, 4, 7<br/>each receives 1/9 = 11.1%"]
C2["Bucket 2<br/>1/3 traffic"] --> C2S["Sinks 2, 5, 8<br/>each receives 1/9 = 11.1%"]
CI["Result: all sinks are reachable,<br/>but expected load is 8.3% or 11.1%"]
end
subgraph WEIGHTED["Proposed algorithm — 30 logical slots"]
direction TB
L["LCM(3 buckets, 10 sinks) = 30 slots<br/>10 slots per bucket · 3 slots per sink"]
W0["Bucket 0 · slots 0–9"] --> W0S["Sink 0 ×3 · Sink 1 ×3<br/>Sink 2 ×3 · Sink 3 ×1"]
W1["Bucket 1 · slots 10–19"] --> W1S["Sink 3 ×2 · Sink 4 ×3<br/>Sink 5 ×3 · Sink 6 ×2"]
W2["Bucket 2 · slots 20–29"] --> W2S["Sink 6 ×1 · Sink 7 ×3<br/>Sink 8 ×3 · Sink 9 ×3"]
WI["Each sink owns exactly 3 slots<br/>Expected load = 3/30 = 10%"]
end
A --> CURRENT
A --> WEIGHTED
```
eg: logicalSlot = bucketStart + hash(normalizedLookupKey) % slotsPerBucket
sinkId = logicalSlot / slotsPerSink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5439740589
Hi @loserwang1024, I have one more question about partitioned tables and would appreciate your opinion.
I checked Paimon's current implementation. Its `BucketIdExtractor` derives only the bucket id, and `BucketShuffleStrategy` is shared by partitioned and non-partitioned tables. When the bucket count is lower than the lookup parallelism, it hashes the complete join key within the bucket's round-robin subtask subset. Thus it uses all subtasks, but one bucket can be cached/accessed by multiple subtasks.
For Fluss, the physical routing unit is a `(partition, bucket)` tablet. The current PR hashes that pair to one lookup subtask, preserving tablet/cache/RPC locality. The trade-off is that when the number of active tablets is lower than the lookup parallelism, some subtasks can remain idle. Before this PR, Flink's regular hash shuffle was already correct and could use the available parallelism, but it did not preserve tablet affinity.
Would you prefer keeping the current `(partition, bucket)` affinity in this PR and leaving adaptive fan-out for the low-tablet case as a follow-up, or aligning partitioned tables with Paimon's bucket-only strategy now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5492318612
Thanks for the review. I have addressed the comments and simplified the implementation:
- Bucket custom shuffle is enabled based on the presence of bucket keys; tables without bucket keys fall back to Flink's default distribution.
- Bucket keys are validated against the normalized lookup-key fields.
- For shuffle routing, existing mappings are preserved when the bucket and subtask counts evenly divide each other. Otherwise, weighted logical slots derived from their GCD are used to balance the expected subtask load while keeping routing deterministic and bucket fan-out bounded. The same strategy applies to partitioned and non-partitioned tables.
- Added unit coverage for both `3 buckets / 10 subtasks` and `10 buckets / 3 subtasks`, and MiniCluster IT coverage for `bucket.num = {1, 3, 8, 10}`. All related tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3871514760
##########
fluss-flink/fluss-flink-2.2/src/test/java/org/apache/fluss/flink/source/FlinkLookupShuffleTableSourceTest.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.source;
+
+import org.apache.fluss.bucketing.BucketingFunction;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.flink.FlinkConnectorOptions;
+import org.apache.fluss.flink.row.FlinkAsFlussRow;
+import org.apache.fluss.flink.utils.FlinkConnectorOptionsUtils;
+import org.apache.fluss.flink.utils.FlinkConversions;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.encode.KeyEncoder;
+import org.apache.fluss.testutils.common.MultiVersionTest;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.LookupTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsLookupCustomShuffle.InputDataPartitioner;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * Unit tests for {@link FlinkLookupShuffleTableSource}, focusing on the invariant that a wrapped
+ * source (which has already advertised {@code SupportsLookupCustomShuffle}, so Flink suppresses its
+ * own hash shuffle) must always return a partitioner, and that a copy made between {@code
+ * getLookupRuntimeProvider()} and {@code getPartitioner()} keeps the stashed lookup normalizer.
+ */
+@MultiVersionTest
+class FlinkLookupShuffleTableSourceTest {
+
+ private static FlinkTableSource baseSource(int[] primaryKeys, int[] bucketKeys) {
+ RowType tableOutputType =
+ (RowType)
+ DataTypes.ROW(
+ DataTypes.FIELD("id", DataTypes.INT()),
+ DataTypes.FIELD("name", DataTypes.STRING()),
+ DataTypes.FIELD("address", DataTypes.STRING()))
+ .getLogicalType();
+ return baseSource(
+ tableOutputType,
+ primaryKeys,
+ bucketKeys,
+ new int[] {},
+ new TableConfig(new Configuration()));
+ }
+
+ private static FlinkTableSource baseSource(
+ RowType tableOutputType,
+ int[] primaryKeys,
+ int[] bucketKeys,
+ int[] partitionKeys,
+ TableConfig tableConfig) {
+ Configuration flussConfig = new Configuration();
+ flussConfig.setString(FlinkConnectorOptions.BOOTSTRAP_SERVERS.key(), "localhost:9092");
+ FlinkConnectorOptionsUtils.StartupOptions startupOptions =
+ new FlinkConnectorOptionsUtils.StartupOptions();
+ startupOptions.startupMode = FlinkConnectorOptions.ScanStartupMode.EARLIEST;
+ return new FlinkTableSource(
+ TablePath.of("test_db", "dim"),
+ flussConfig,
+ tableConfig,
+ tableOutputType,
+ primaryKeys,
+ bucketKeys,
+ partitionKeys,
+ true, // streaming
+ startupOptions,
+ false, // lookup async (sync avoids any async runtime setup)
+ false, // insert if not exists
+ null, // cache
+ 1000L, // scan partition discovery interval
+ false, // is data lake enabled
+ null, // merge engine type
+ new HashMap<>(),
+ null); // lease context
+ }
+
+ @Test
+ void testGetPartitionerPresentForFullPkLookup() {
+ // primary key == bucket key == id; lookup on the full primary key.
+ FlinkLookupShuffleTableSource source =
+ new FlinkLookupShuffleTableSource(baseSource(new int[] {0}, new int[] {0}), 3);
+ source.getLookupRuntimeProvider(new TestLookupContext(new int[][] {{0}}));
+ assertThat(source.getPartitioner()).isPresent();
+ }
+
+ @Test
+ void testPartitionedLakeFormatBuildsUsablePartitioner() {
+ RowType tableOutputType =
+ (RowType)
+ DataTypes.ROW(
+ DataTypes.FIELD("id", DataTypes.INT().notNull()),
+ DataTypes.FIELD("region", DataTypes.STRING().notNull()),
+ DataTypes.FIELD("p_date", DataTypes.STRING().notNull()))
+ .getLogicalType();
+ Configuration tableConfiguration = new Configuration();
+ tableConfiguration.set(ConfigOptions.TABLE_DATALAKE_FORMAT, DataLakeFormat.ICEBERG);
+ FlinkLookupShuffleTableSource source =
+ new FlinkLookupShuffleTableSource(
+ baseSource(
+ tableOutputType,
+ new int[] {0, 1, 2},
+ new int[] {0},
+ new int[] {1, 2},
+ new TableConfig(tableConfiguration)),
+ 16);
+ source.getLookupRuntimeProvider(new TestLookupContext(new int[][] {{0}, {1}, {2}}));
+
+ InputDataPartitioner partitioner = source.getPartitioner().get();
+ RowData first =
+ GenericRowData.of(
+ 1, StringData.fromString("cn"), StringData.fromString("2026-08-20"));
+ RowData second =
+ GenericRowData.of(
+ 2, StringData.fromString("cn"), StringData.fromString("2026-08-20"));
+
+ // The golden bucket vectors pin both ids to Iceberg bucket 4.
+ assertThat(partitioner.partition(second, 4)).isEqualTo(partitioner.partition(first, 4));
+ }
+
+ @Test
+ void testGetPartitionerPresentForPrefixLookup() {
Review Comment:
Thanks, agreed. The previous tests were mostly Flink 2.2-specific implementation tests and did not sufficiently verify the real SQL runtime behavior.
I removed the redundant tests and moved the IT coverage into FlinkTableSourceITCase. The test now runs a real SQL lookup join on a Flink MiniCluster, checks explainSql, and verifies the actual lookup subtask routing for both bucket.num < parallelism and bucket.num >= parallelism. The Flink 2.2 subclass only enables this test capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5519888433
Thanks @leonardBang for pointing this out. I confirmed that the raw DDL path you mentioned is reachable and updated the implementation accordingly. When `bucket.num` is missing or non-positive, the source now returns an empty partitioner, allowing regular lookup joins to continue and custom shuffle to fall back to Flink’s default distribution.
I kept the bucket-key/lookup-key validation unchanged because valid full-PK and prefix lookups are already normalized to include all bucket keys. I also added regression coverage for the fallback behavior on Flink 1.20 and 2.2 and clarified it in the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3871511393
##########
fluss-flink/fluss-flink-2.2/src/main/java/org/apache/fluss/flink/adapter/LookupShuffleSourceAdapter.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.adapter;
+
+import org.apache.fluss.flink.FlinkConnectorOptions;
+import org.apache.fluss.flink.source.FlinkLookupShuffleTableSource;
+import org.apache.fluss.flink.source.FlinkTableSource;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+/**
+ * Flink 2.x implementation: wraps the source so it implements {@code SupportsLookupCustomShuffle},
+ * but only when the table exposes the metadata required to reproduce Fluss bucket routing. Shadows
+ * the common (no-op) class of the same fully-qualified name at package time.
+ *
+ * <p>Eligibility is decided here, at planning time, rather than inside {@code getPartitioner()}:
+ *
+ * <ul>
+ * <li>the table is not lookup-shuffle eligible, or {@code bucket.num} is not configured ->
+ * leave the source unwrapped so that an explicitly requested lookup shuffle falls back to
+ * Flink's default hash distribution;
+ * <li>{@code bucket.num} configured but not a positive integer -> fail fast with a {@link
+ * ValidationException} (a real misconfiguration);
+ * <li>otherwise wrap with the validated bucket count, so the wrapped source can always return a
+ * partitioner.
+ * </ul>
+ *
+ * <p>The unwrapped (hash-fallback) branch is a defensive safety net rather than a reachable state
+ * for a Fluss table: a Fluss table can only be created through the {@code FlussCatalog}, which
+ * always defaults {@code bucket.num} and derives the bucket key from the primary key, while Flink
+ * rejects creating a Fluss source as a temporary/{@code connector}-based table. It is therefore
+ * covered by adapter unit tests rather than an end-to-end planner test.
+ */
+public class LookupShuffleSourceAdapter {
Review Comment:
Thanks for the suggestion. The previous implementation incorrectly treated custom shuffle as a capability that needed factory-level gating and fallback handling, which introduced an extra table source wrapper and copy/state-preservation logic. I also overcomplicated the case where the bucket count is smaller than the lookup parallelism.
I have changed it to follow the same-FQN adapter pattern: `FlinkTableSource` directly implements `SupportsLookupCustomShuffleAdapter`, the Flink 2.2 adapter extends `SupportsLookupCustomShuffle`, and the factory returns `FlinkTableSource` directly.
For non-partitioned tables, the revised routing is:
- If `numBuckets >= numPartitions`, route by `bucketId % numPartitions`.
- If `numBuckets < numPartitions`, assign each bucket a disjoint round-robin subset of subtasks and hash the normalized lookup key within that subset. This keeps the same key on a stable subtask while allowing all lookup subtasks to be utilized instead of leaving the extra subtasks idle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5512082054
Thanks for the thorough work here — the routing algorithm (divisible / round-robin / LCM weighted-slot branches), the frozen golden vectors across lake formats, and the MiniCluster ITCase that compares the real runtime subtask against the production partitioner are all solid. One thing I'd like you to double-check before merging.
**`createLookupInputPartitioner` runs for every lookup join and hard-fails when `bucket.num` can't be resolved, even when the `shuffle=true` hint is absent.**
In `FlinkTableSource#getLookupRuntimeProvider`, `this.lookupInputPartitioner = createLookupInputPartitioner(lookupNormalizer)` is invoked unconditionally. As soon as `bucketKeyIndexes.length > 0`, it calls `checkNotNull(tableOptions.get(BUCKET_NUMBER), ...)`, and the `FlussLookupInputPartitioner` constructor additionally does `checkArgument(bucketKeys ⊆ lookupKeys)`.
For tables resolved through the Fluss Catalog this is safe, since `bucket.num` is always injected (`FlinkConversions` puts `tableInfo.getNumBuckets()` into the options). But for a Fluss table declared via raw DDL (`connector = 'fluss'` in a non-Fluss catalog) that sets `bucket.key` without `bucket.num`, `tableOptions` won't contain `BUCKET_NUMBER`, so `checkNotNull` throws and the lookup job fails at plan/build time — even though the user never requested custom shuffle. This seems to diverge from the stated intent of "fail explicitly **when custom shuffle is requested**": today it effectively fails for *any* lookup on a bucket-keyed table whose `bucket.num` isn't statically present.
Could you confirm whether that DDL path is actually reachable/supported? If it is, I'd suggest making it fail-soft — return `null` (i.e. `Optional.empty()`) when `bucket.num` is missing or the bucket keys aren't a subset of the lookup keys — so the optimization degrades to Flink's default distribution instead of regressing an otherwise-working lookup join. Alternatively, gate the construction on `LookupContext#preferCustomShuffle()` (available on Flink 2.2) and keep the fail-fast only for the case where the shuffle was actually requested.
Note this is about robustness, not join correctness: if a DDL-declared `bucket.num` differs from the real server-side bucket count, only the locality optimization degrades — each subtask's lookuper still routes to the correct bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3869364515
##########
fluss-flink/fluss-flink-2.2/src/main/java/org/apache/fluss/flink/adapter/LookupShuffleSourceAdapter.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.adapter;
+
+import org.apache.fluss.flink.FlinkConnectorOptions;
+import org.apache.fluss.flink.source.FlinkLookupShuffleTableSource;
+import org.apache.fluss.flink.source.FlinkTableSource;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.ValidationException;
+
+/**
+ * Flink 2.x implementation: wraps the source so it implements {@code SupportsLookupCustomShuffle},
+ * but only when the table exposes the metadata required to reproduce Fluss bucket routing. Shadows
+ * the common (no-op) class of the same fully-qualified name at package time.
+ *
+ * <p>Eligibility is decided here, at planning time, rather than inside {@code getPartitioner()}:
+ *
+ * <ul>
+ * <li>the table is not lookup-shuffle eligible, or {@code bucket.num} is not configured ->
+ * leave the source unwrapped so that an explicitly requested lookup shuffle falls back to
+ * Flink's default hash distribution;
+ * <li>{@code bucket.num} configured but not a positive integer -> fail fast with a {@link
+ * ValidationException} (a real misconfiguration);
+ * <li>otherwise wrap with the validated bucket count, so the wrapped source can always return a
+ * partitioner.
+ * </ul>
+ *
+ * <p>The unwrapped (hash-fallback) branch is a defensive safety net rather than a reachable state
+ * for a Fluss table: a Fluss table can only be created through the {@code FlussCatalog}, which
+ * always defaults {@code bucket.num} and derives the bucket key from the primary key, while Flink
+ * rejects creating a Fluss source as a temporary/{@code connector}-based table. It is therefore
+ * covered by adapter unit tests rather than an end-to-end planner test.
+ */
+public class LookupShuffleSourceAdapter {
Review Comment:
The current `LookupShuffleSourceAdapter` seems more complicated than necessary. It performs capability gating in the factory, wraps `FlinkTableSource` with an additional subtype, and requires extra copy/state-preservation logic.
Could we follow the existing version-specific superclass adapter pattern instead? For example, let `FlinkTableSource` extend a same-FQN `SupportsLookupCustomShuffleAdapter`.
* The Flink 2.2 implementation can implement `SupportsLookupCustomShuffle` and provide `getPartitioner()`
```java
public interface SupportsLookupCustomShuffleAdapter extends SupportsLookupCustomShuffle {
@Override
default Optional<InputDataPartitioner> getPartitioner() {
Optional<InputDataPartitionerAdapter> partitionerAdapter = getPartitionerAdapter();
Optional<InputDataPartitioner> inputDataPartitioner = partitionerAdapter.isPresent() ? Optional.of(partitionerAdapter.get()) : Optional.empty();
return inputDataPartitioner;
}
Optional<InputDataPartitionerAdapter> getPartitionerAdapter();
interface InputDataPartitionerAdapter extends InputDataPartitioner{}
}
```
* the lower-version implementation :
```java
public interface SupportsLookupCustomShuffleAdapter {
Optional<InputDataPartitionerAdapter> getPartitionerAdapter();
interface InputDataPartitionerAdapter {
int partition(RowData joinKeys, int numPartitions);
default boolean isDeterministic() {
return true;
}
}
}
```
##########
fluss-flink/fluss-flink-2.2/src/test/java/org/apache/fluss/flink/source/FlinkLookupShuffleTableSourceTest.java:
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.source;
+
+import org.apache.fluss.bucketing.BucketingFunction;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+import org.apache.fluss.config.TableConfig;
+import org.apache.fluss.flink.FlinkConnectorOptions;
+import org.apache.fluss.flink.row.FlinkAsFlussRow;
+import org.apache.fluss.flink.utils.FlinkConnectorOptionsUtils;
+import org.apache.fluss.flink.utils.FlinkConversions;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.metadata.TablePath;
+import org.apache.fluss.row.encode.KeyEncoder;
+import org.apache.fluss.testutils.common.MultiVersionTest;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.connector.source.LookupTableSource;
+import org.apache.flink.table.connector.source.abilities.SupportsLookupCustomShuffle.InputDataPartitioner;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Optional;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/**
+ * Unit tests for {@link FlinkLookupShuffleTableSource}, focusing on the invariant that a wrapped
+ * source (which has already advertised {@code SupportsLookupCustomShuffle}, so Flink suppresses its
+ * own hash shuffle) must always return a partitioner, and that a copy made between {@code
+ * getLookupRuntimeProvider()} and {@code getPartitioner()} keeps the stashed lookup normalizer.
+ */
+@MultiVersionTest
+class FlinkLookupShuffleTableSourceTest {
+
+ private static FlinkTableSource baseSource(int[] primaryKeys, int[] bucketKeys) {
+ RowType tableOutputType =
+ (RowType)
+ DataTypes.ROW(
+ DataTypes.FIELD("id", DataTypes.INT()),
+ DataTypes.FIELD("name", DataTypes.STRING()),
+ DataTypes.FIELD("address", DataTypes.STRING()))
+ .getLogicalType();
+ return baseSource(
+ tableOutputType,
+ primaryKeys,
+ bucketKeys,
+ new int[] {},
+ new TableConfig(new Configuration()));
+ }
+
+ private static FlinkTableSource baseSource(
+ RowType tableOutputType,
+ int[] primaryKeys,
+ int[] bucketKeys,
+ int[] partitionKeys,
+ TableConfig tableConfig) {
+ Configuration flussConfig = new Configuration();
+ flussConfig.setString(FlinkConnectorOptions.BOOTSTRAP_SERVERS.key(), "localhost:9092");
+ FlinkConnectorOptionsUtils.StartupOptions startupOptions =
+ new FlinkConnectorOptionsUtils.StartupOptions();
+ startupOptions.startupMode = FlinkConnectorOptions.ScanStartupMode.EARLIEST;
+ return new FlinkTableSource(
+ TablePath.of("test_db", "dim"),
+ flussConfig,
+ tableConfig,
+ tableOutputType,
+ primaryKeys,
+ bucketKeys,
+ partitionKeys,
+ true, // streaming
+ startupOptions,
+ false, // lookup async (sync avoids any async runtime setup)
+ false, // insert if not exists
+ null, // cache
+ 1000L, // scan partition discovery interval
+ false, // is data lake enabled
+ null, // merge engine type
+ new HashMap<>(),
+ null); // lease context
+ }
+
+ @Test
+ void testGetPartitionerPresentForFullPkLookup() {
+ // primary key == bucket key == id; lookup on the full primary key.
+ FlinkLookupShuffleTableSource source =
+ new FlinkLookupShuffleTableSource(baseSource(new int[] {0}, new int[] {0}), 3);
+ source.getLookupRuntimeProvider(new TestLookupContext(new int[][] {{0}}));
+ assertThat(source.getPartitioner()).isPresent();
+ }
+
+ @Test
+ void testPartitionedLakeFormatBuildsUsablePartitioner() {
+ RowType tableOutputType =
+ (RowType)
+ DataTypes.ROW(
+ DataTypes.FIELD("id", DataTypes.INT().notNull()),
+ DataTypes.FIELD("region", DataTypes.STRING().notNull()),
+ DataTypes.FIELD("p_date", DataTypes.STRING().notNull()))
+ .getLogicalType();
+ Configuration tableConfiguration = new Configuration();
+ tableConfiguration.set(ConfigOptions.TABLE_DATALAKE_FORMAT, DataLakeFormat.ICEBERG);
+ FlinkLookupShuffleTableSource source =
+ new FlinkLookupShuffleTableSource(
+ baseSource(
+ tableOutputType,
+ new int[] {0, 1, 2},
+ new int[] {0},
+ new int[] {1, 2},
+ new TableConfig(tableConfiguration)),
+ 16);
+ source.getLookupRuntimeProvider(new TestLookupContext(new int[][] {{0}, {1}, {2}}));
+
+ InputDataPartitioner partitioner = source.getPartitioner().get();
+ RowData first =
+ GenericRowData.of(
+ 1, StringData.fromString("cn"), StringData.fromString("2026-08-20"));
+ RowData second =
+ GenericRowData.of(
+ 2, StringData.fromString("cn"), StringData.fromString("2026-08-20"));
+
+ // The golden bucket vectors pin both ids to Iceberg bucket 4.
+ assertThat(partitioner.partition(second, 4)).isEqualTo(partitioner.partition(first, 4));
+ }
+
+ @Test
+ void testGetPartitionerPresentForPrefixLookup() {
Review Comment:
I don't like this test.
1. ItCase need to be add to flink-common module.
2. An IT case test means end to end, please add flink sql tests for ITCase(Adding test case in FlinkTableSourceITCase is enough, referring org.apache.fluss.flink.source.FlinkTableSourceITCase#testLookup1PkTable)
```sql
SELECT /*+ LOOKUP('table'='Customers', 'shuffle'='true') */ o.order_id, o.total, c.country, c.zip
FROM Orders AS o
JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = [c.id](http://c.id/);
```
3. To verify shuffle is enbaled, check the result of tevn.explainSql("xxx")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "loserwang1024 (via GitHub)" <gi...@apache.org>.
loserwang1024 merged PR #3675:
URL: https://github.com/apache/fluss/pull/3675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5598894171
Thanks @leonardBang, and sorry for the confusion in my earlier reply. Raw connector DDL outside the Fluss Catalog currently fails at connector discovery, before lookup setup; I verified this with lookup SQL planning checks both with and without `bucket.num` and the shuffle hint. The supported Fluss Catalog path supplies the resolved bucket count even when it is omitted from the DDL. I've therefore restored the required bucket-count check and removed the fallback test and documentation for the unsupported path in 3438ceceb. For the bucket-key subset concern, the existing `LookupNormalizer` rejects lookups missing required bucket keys before the partitioner is constructed. This validation predates this PR, so keeping the partitioner's check does not reject an otherwise-supported lookup. I also re-ran the existing missing-key SQL tests and checked full-PK and prefix normalization with projection. Could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on code in PR #3675:
URL: https://github.com/apache/fluss/pull/3675#discussion_r3903007049
##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/source/lookup/FlussLookupInputPartitioner.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.source.lookup;
+
+import org.apache.fluss.bucketing.BucketingFunction;
+import org.apache.fluss.flink.adapter.SupportsLookupCustomShuffleAdapter.InputDataPartitionerAdapter;
+import org.apache.fluss.flink.row.FlinkAsFlussRow;
+import org.apache.fluss.flink.utils.FlinkConversions;
+import org.apache.fluss.metadata.DataLakeFormat;
+import org.apache.fluss.row.InternalRow;
+import org.apache.fluss.row.encode.CompactedKeyEncoder;
+import org.apache.fluss.row.encode.KeyEncoder;
+import org.apache.fluss.utils.MathUtils;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.logical.RowType;
+
+import javax.annotation.Nullable;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.fluss.utils.Preconditions.checkArgument;
+
+/**
+ * Partitions the lookup-join probe stream by Fluss bucket, consistent with the client-side
+ * bucketing used by {@code PrimaryKeyLookuper}/{@code PrefixKeyLookuper} (bucket-key encoding +
+ * {@link BucketingFunction}).
+ *
+ * <p>Partitioned and non-partitioned tables use the same strategy. When there are fewer buckets
+ * than lookup subtasks, each bucket is assigned a subset of subtasks and normalized lookup keys are
+ * hashed within that subset. This avoids leaving lookup subtasks idle for small-bucket tables while
+ * keeping every lookup key on a stable subtask and bounding each bucket's RPC fan-out.
+ */
+public class FlussLookupInputPartitioner implements InputDataPartitionerAdapter {
+
+ private static final long serialVersionUID = 1L;
+
+ private final LookupNormalizer normalizer;
+ // Flink row type of the normalized lookup key row (the expected lookup keys in Fluss key order,
+ // i.e. the full primary key for a primary-key lookup, or the bucket keys + partition keys for a
+ // prefix lookup).
+ private final RowType keyFlinkRowType;
+ private final List<String> bucketKeyNames;
+ @Nullable private final DataLakeFormat lakeFormat;
+ private final int numBuckets;
+
+ private transient KeyEncoder bucketKeyEncoder;
+ private transient KeyEncoder lookupKeyEncoder;
+ private transient BucketingFunction bucketingFunction;
+ private transient FlinkAsFlussRow reuseRow;
+
+ /**
+ * Creates a partitioner consistent with Fluss client-side bucket routing.
+ *
+ * @param normalizer normalizes Flink lookup keys into Fluss lookup-key order
+ * @param keyFlinkRowType row type of the normalized lookup key
+ * @param bucketKeyNames bucket-key field names within the normalized lookup key
+ * @param lakeFormat optional lake format that defines key encoding and bucketing behavior
+ * @param numBuckets positive number of buckets in the Fluss table
+ */
+ public FlussLookupInputPartitioner(
+ LookupNormalizer normalizer,
+ RowType keyFlinkRowType,
+ List<String> bucketKeyNames,
+ @Nullable DataLakeFormat lakeFormat,
+ int numBuckets) {
+ this.normalizer = normalizer;
+ this.keyFlinkRowType = keyFlinkRowType;
+ this.bucketKeyNames = bucketKeyNames;
+ this.lakeFormat = lakeFormat;
+ checkArgument(numBuckets > 0, "numBuckets must be positive, but was %s.", numBuckets);
Review Comment:
Added. The partitioner now validates that bucket keys are non-empty and are a subset of the normalized lookup-key fields, consistent with LookupNormalizer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "naivedogger (via GitHub)" <gi...@apache.org>.
naivedogger commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5000659039
Thanks @zuston for your comment!
For some context: I'm trying to help enhance Fluss's Flink connector, and this feature is similar to Paimon's [Large Scale Lookup (Fixed Bucket)](https://paimon.apache.org/docs/master/flink/sql-lookup/#large-scale-lookup-fixed-bucket) — it only kicks in on Flink 2.0+ and when the join keys contain all bucket keys.
**What shuffle lookup buys us:** rows of the same bucket are routed to the same lookup subtask, so each subtask only deals with its own buckets. This way, when partial caching is enabled, each subtask only needs to cache its own slice of the table, giving a potentially better cache hit ratio and less duplicated cached data across subtasks.
Here the cache refers to [Flink's built-in lookup caching](https://nightlies.apache.org/flink/flink-docs-stable/docs/sql/reference/queries/hints/), not a Fluss-specific one — that's probably why it was hard to spot. `FULL` caching isn't supported yet; I'm planning to add it as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support custom lookup shuffle by bucket for Flink 2.x [fluss]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #3675:
URL: https://github.com/apache/fluss/pull/3675#issuecomment-5000736438
Thanks for your detailed explanation, @naivedogger. Much appreciated. 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org