You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/08/02 11:32:42 UTC

Re: [PR] [lake/paimon] Support custom Paimon lake table path [fluss]

luoyuxia commented on code in PR #3476:
URL: https://github.com/apache/fluss/pull/3476#discussion_r3698584660


##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/LakeEnabledTableCreateITCase.java:
##########
@@ -1055,6 +1059,234 @@ void testEnableLakeTableAfterAlterTableProperties() throws Exception {
         admin.alterTable(tablePath, Collections.singletonList(enableLake), false).get();
     }
 
+    @Test
+    void testAlterLakePathOptionsValidation() throws Exception {

Review Comment:
   If these mapping options are create-time only as discussed above, this test is exercising a state machine that we should not support: altering the path while lake is disabled, allowing changes when the resolved path is unchanged, and gating changes on tiering progress. Could we replace it with a small test asserting that SET/RESET of both mapping options is always rejected after table creation?



##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java:
##########
@@ -577,6 +580,29 @@ public void alterTableProperties(
         }
     }
 
+    private void validateAlterLakeTablePath(TableInfo tableInfo, TableDescriptor newDescriptor)

Review Comment:
   Based on the discussion above, my understanding is that these two mapping options should be create-time only. However, the current implementation still allows changing them when datalake is disabled and no tiering progress exists, and also allows option changes when the resolved path remains unchanged.
   
   Could we remove these options from ALTERABLE_TABLE_OPTIONS and drop validateAlterLakeTablePath() entirely? That would match the agreed semantics and avoid the additional lifecycle, migration, and in-flight tiering complexity.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/LakeTableFactory.java:
##########
@@ -26,23 +28,30 @@
 import org.apache.flink.table.factories.FactoryUtil;
 
 import java.util.Collections;
+import java.util.Map;
 
 /** A factory to create {@link DynamicTableSource} for lake table. */
 public class LakeTableFactory {
+
+    private static final String FLUSS_CONF_PREFIX = "fluss.";
+    private static final String FLUSS_TABLE_DATALAKE_DATABASE_NAME =
+            FLUSS_CONF_PREFIX + ConfigOptions.TABLE_DATALAKE_DATABASE_NAME.key();
+    private static final String FLUSS_TABLE_DATALAKE_TABLE_NAME =
+            FLUSS_CONF_PREFIX + ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key();
+
     private final LakeFlinkCatalog lakeFlinkCatalog;
 
     public LakeTableFactory(LakeFlinkCatalog lakeFlinkCatalog) {
         this.lakeFlinkCatalog = lakeFlinkCatalog;
     }
 
     public DynamicTableSource createDynamicTableSource(
-            DynamicTableFactory.Context context, String tableName) {
-        ObjectIdentifier originIdentifier = context.getObjectIdentifier();
+            DynamicTableFactory.Context context, String requestedTableName) {

Review Comment:
   I’m a bit confused by the rename from `tableName` to `requestedTableName`; the new name is not very intuitive to me.
   
   As I understand it, this value is the original `tableName` after `$lake` has been removed:
   
   ```text
   fluss_table$lake$snapshots
       -> fluss_table$snapshots
   ```
   
   It is neither the complete name requested by the user nor the actual lake table name after applying the custom mapping. Could we keep the original `tableName`, or use a more precise name such as `logicalLakeObjectName`?



##########
fluss-common/src/main/java/org/apache/fluss/metadata/LakeTableUtil.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.metadata;
+
+import org.apache.fluss.annotation.Internal;
+import org.apache.fluss.config.ConfigOptions;
+import org.apache.fluss.config.Configuration;
+
+import java.util.Map;
+import java.util.Optional;
+
+/** Utility methods for resolving external lake table metadata. */
+@Internal
+public final class LakeTableUtil {
+
+    private LakeTableUtil() {}
+
+    /** Returns the table path used to access the external datalake table. */
+    public static TablePath getLakeTablePath(

Review Comment:
   Could we simplify the path API to expose a single resolver that accepts `ReadableConfig`?
   
   ```java
   public static TablePath resolveLakeTablePath(
           TablePath flussTablePath, ReadableConfig tableConfig)
   ```
   
   Both `Configuration` and the table properties held by `TableInfo` can use this API, so the `Map`/`Configuration` overloads and the private optional getters are not needed.
   
   I also find `hasCustomLakePath()` ambiguous: it currently checks whether either option is present, not whether the resolved lake path actually differs from the Fluss path. An explicit identity mapping would therefore be reported as custom. Callers can express the intended semantics directly: use `Configuration#contains(...)` for option-presence validation, and compare the resolved path with the Fluss path when checking for an actual path change.
   
   I tried this approach, including the affected call sites and a focused test, in [ac6cd522f](https://github.com/luoyuxia/fluss/commit/ac6cd522f1c161e802a354c1b15f2875796eb1d1) for reference.



##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeUtils.scala:
##########
@@ -52,6 +52,10 @@ object FlussLakeUtils extends Logging {
       catalogProperties: util.Map[String, String],
       tableProperties: util.Map[String, String],
       tablePath: TablePath): LakeSource[LakeSplit] = {
+    if (LakeTableUtil.hasCustomLakePath(tableProperties)) {
+      throw new UnsupportedOperationException(

Review Comment:
   please add todo comment and create a issue for it.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/LakeTableFactory.java:
##########
@@ -26,23 +28,30 @@
 import org.apache.flink.table.factories.FactoryUtil;
 
 import java.util.Collections;
+import java.util.Map;
 
 /** A factory to create {@link DynamicTableSource} for lake table. */
 public class LakeTableFactory {
+
+    private static final String FLUSS_CONF_PREFIX = "fluss.";

Review Comment:
   This dependency on the persisted `fluss.*` options feels implicit and fragile. `LakeTableFactory` assumes that every lake format persists the original Fluss options using this exact prefix. If we later stop persisting these options, filter them out, or change the prefix, custom lake path resolution will stop working and may silently fall back to the original Fluss path.
   
   Could we resolve the final lake identifier once in `FlinkCatalog`, where `TableInfo#getLakeTablePath()` and the metadata-table suffix are already available, and attach it as transient metadata to the returned `CatalogTable` copy?
   
   For example:
   
   ```java
   Map<String, String> options = new HashMap<>(lakeTable.getOptions());
   options.put(RESOLVED_LAKE_DATABASE, lakeTablePath.getDatabaseName());
   options.put(RESOLVED_LAKE_OBJECT, lakeObjectName);
   return lakeTable.copy(options);
   ```
   
   `LakeTableFactory` can then consume and remove these values before delegating to the Paimon/Iceberg factory:
   
   ```java
   Map<String, String> options =
           new HashMap<>(context.getCatalogTable().getOptions());
   
   String lakeDatabase = checkNotNull(options.remove(RESOLVED_LAKE_DATABASE));
   String lakeObject = checkNotNull(options.remove(RESOLVED_LAKE_OBJECT));
   
   ObjectIdentifier lakeIdentifier =
           ObjectIdentifier.of(
                   context.getObjectIdentifier().getCatalogName(),
                   lakeDatabase,
                   lakeObject);
   ```
   
   These metadata values would exist only on the in-memory table returned to the Flink planner and would be removed before creating the delegated factory context. This avoids another metadata RPC while removing the dependency on the persisted `fluss.*` properties.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/catalog/FlinkCatalog.java:
##########
@@ -469,23 +471,19 @@ public CatalogBaseTable getTable(ObjectPath objectPath)
     }
 
     protected CatalogBaseTable getLakeTable(
-            String databaseName,
-            String tableName,
+            TablePath lakeTablePath,
+            String requestedTableName,

Review Comment:
   I find `requestedTableName` difficult to follow because it represents the original Flink catalog object name (for example, `fluss_table$lake$snapshots`), while this method ultimately accesses a resolved lake object such as `custom_lake_table$snapshots`.
   
   Could we resolve the lake object name at the call site and let `getLakeTable()` only load the resolved object? For example:
   
   ```java
   String lakeObjectName =
           LakeTableUtil.getLakeTableName(
                   lakeTablePath.getTableName(), tableName, LAKE_TABLE_SPLITTER);
   
   return getLakeTable(
           lakeTablePath.getDatabaseName(),
           lakeObjectName,
           tableInfo.getProperties(),
           getLakeCatalogProperties());
   ```
   
   Then the method can accept `lakeDatabaseName` and `lakeObjectName` directly. This keeps the `$lake`/metadata-table translation at the request-handling layer and avoids passing the ambiguous `requestedTableName` through another layer.



##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/PaimonTieringITCase.java:
##########
@@ -188,6 +188,32 @@ void testTiering() throws Exception {
         }
     }
 
+    @Test
+    void testTieringWithCustomLakeTablePath() throws Exception {

Review Comment:
   This seems redundant with `testUnionReadWithCustomLakeTablePath`, which already starts the tiering job and verifies that data is written to and read from the mapped Paimon table. The additional snapshot-offset assertion here covers generic tiering behavior that is already exercised by `testTiering`. Could we remove this test?



##########
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/LakeEnabledTableCreateITCase.java:
##########
@@ -1055,6 +1059,234 @@ void testEnableLakeTableAfterAlterTableProperties() throws Exception {
         admin.alterTable(tablePath, Collections.singletonList(enableLake), false).get();
     }
 
+    @Test
+    void testAlterLakePathOptionsValidation() throws Exception {
+        TablePath lakeDisabledTablePath = TablePath.of(DATABASE, "lake_path_disabled");
+        Map<String, String> disabledProperties = new HashMap<>();
+        disabledProperties.put(ConfigOptions.TABLE_DATALAKE_ENABLED.key(), "false");
+        disabledProperties.put(ConfigOptions.TABLE_DATALAKE_DATABASE_NAME.key(), "lake_db");
+        TableDescriptor disabledTableDescriptor =
+                TableDescriptor.builder()
+                        .schema(
+                                Schema.newBuilder()
+                                        .column("c1", DataTypes.INT())
+                                        .column("c2", DataTypes.STRING())
+                                        .build())
+                        .properties(disabledProperties)
+                        .distributedBy(BUCKET_NUM, "c1", "c2")
+                        .build();
+        admin.createTable(lakeDisabledTablePath, disabledTableDescriptor, false).get();
+
+        admin.alterTable(
+                        lakeDisabledTablePath,
+                        Collections.singletonList(
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(),
+                                        "lake_table")),
+                        false)
+                .get();
+
+        TableInfo disabledTable = admin.getTableInfo(lakeDisabledTablePath).get();
+        assertThat(disabledTable.getProperties().toMap())
+                .containsEntry(ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(), "lake_table");
+        assertThatThrownBy(
+                        () ->
+                                paimonCatalog.getTable(
+                                        Identifier.create(
+                                                DATABASE, lakeDisabledTablePath.getTableName())))
+                .isInstanceOf(Catalog.TableNotExistException.class);
+
+        admin.alterTable(
+                        lakeDisabledTablePath,
+                        Collections.singletonList(
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_ENABLED.key(), "true")),
+                        false)
+                .get();
+        FileStoreTable paimonTable =
+                (FileStoreTable) paimonCatalog.getTable(Identifier.create("lake_db", "lake_table"));
+        assertThat(paimonTable.options())
+                .containsEntry(
+                        "fluss." + ConfigOptions.TABLE_DATALAKE_DATABASE_NAME.key(), "lake_db")
+                .containsEntry(
+                        "fluss." + ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(), "lake_table");
+        Map<String, String> enabledDisabledTableProperties = new HashMap<>(disabledProperties);
+        enabledDisabledTableProperties.put(
+                ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(), "lake_table");
+        enabledDisabledTableProperties.put(ConfigOptions.TABLE_DATALAKE_ENABLED.key(), "true");
+        verifyPaimonTable(
+                paimonTable,
+                disabledTableDescriptor.withProperties(enabledDisabledTableProperties),
+                RowType.of(
+                        new DataType[] {
+                            org.apache.paimon.types.DataTypes.INT(),
+                            org.apache.paimon.types.DataTypes.STRING(),
+                            org.apache.paimon.types.DataTypes.INT(),
+                            org.apache.paimon.types.DataTypes.BIGINT(),
+                            org.apache.paimon.types.DataTypes.TIMESTAMP_LTZ_MILLIS()
+                        },
+                        new String[] {
+                            "c1",
+                            "c2",
+                            BUCKET_COLUMN_NAME,
+                            OFFSET_COLUMN_NAME,
+                            TIMESTAMP_COLUMN_NAME
+                        }),
+                "c1,c2",
+                BUCKET_NUM);
+
+        TablePath lakeEnabledTablePath = TablePath.of(DATABASE, "lake_path_enabled");
+        Map<String, String> enabledProperties = new HashMap<>();
+        enabledProperties.put(ConfigOptions.TABLE_DATALAKE_ENABLED.key(), "true");
+        enabledProperties.put(ConfigOptions.TABLE_DATALAKE_DATABASE_NAME.key(), "lake_db");
+        enabledProperties.put(ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(), "lake_table_enabled");
+        TableDescriptor enabledTableDescriptor =
+                TableDescriptor.builder()
+                        .schema(
+                                Schema.newBuilder()
+                                        .column("c1", DataTypes.INT())
+                                        .column("c2", DataTypes.STRING())
+                                        .build())
+                        .properties(enabledProperties)
+                        .distributedBy(BUCKET_NUM, "c1", "c2")
+                        .build();
+        admin.createTable(lakeEnabledTablePath, enabledTableDescriptor, false).get();
+        FileStoreTable enabledPaimonTable =
+                (FileStoreTable)
+                        paimonCatalog.getTable(Identifier.create("lake_db", "lake_table_enabled"));
+        long schemaId = enabledPaimonTable.schema().id();
+
+        admin.alterTable(
+                        lakeEnabledTablePath,
+                        Arrays.asList(
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_DATABASE_NAME.key(),
+                                        "lake_db"),
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(),
+                                        "lake_table_enabled"),
+                                TableChange.set("key", "value")),
+                        false)
+                .get();
+        enabledPaimonTable =
+                (FileStoreTable)
+                        paimonCatalog.getTable(Identifier.create("lake_db", "lake_table_enabled"));
+        assertThat(enabledPaimonTable.schema().id()).isGreaterThan(schemaId);
+        assertThat(enabledPaimonTable.options()).containsEntry("fluss.key", "value");
+
+        assertThatThrownBy(
+                        () ->
+                                admin.alterTable(
+                                                lakeEnabledTablePath,
+                                                Collections.singletonList(
+                                                        TableChange.set(
+                                                                ConfigOptions
+                                                                        .TABLE_DATALAKE_TABLE_NAME
+                                                                        .key(),
+                                                                "another_lake_table")),
+                                                false)
+                                        .get())
+                .cause()
+                .isInstanceOf(InvalidAlterTableException.class)
+                .hasMessageContaining(
+                        "cannot be altered when 'table.datalake.enabled' is true or the table has tiering progress")
+                .hasMessageContaining(ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key());
+
+        admin.alterTable(
+                        lakeEnabledTablePath,
+                        Collections.singletonList(
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_ENABLED.key(), "false")),
+                        false)
+                .get();
+
+        admin.alterTable(
+                        lakeEnabledTablePath,
+                        Collections.singletonList(
+                                TableChange.set(
+                                        ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(),
+                                        lakeEnabledTablePath.getTableName())),
+                        false)
+                .get();
+        TableInfo disabledEnabledTable = admin.getTableInfo(lakeEnabledTablePath).get();
+        assertThat(disabledEnabledTable.getProperties().toMap())
+                .containsEntry(
+                        ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key(),
+                        lakeEnabledTablePath.getTableName());
+
+        FLUSS_CLUSTER_EXTENSION
+                .getZooKeeperClient()
+                .upsertLakeTable(
+                        disabledEnabledTable.getTableId(),
+                        new LakeTable(
+                                new LakeTable.LakeSnapshotMetadata(
+                                        1L,
+                                        new FsPath(
+                                                FLUSS_CLUSTER_EXTENSION.getRemoteDataDir(),
+                                                "lake-path-progress"),
+                                        null)),
+                        false);
+
+        admin.alterTable(
+                        lakeEnabledTablePath,
+                        Arrays.asList(
+                                TableChange.reset(ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key()),
+                                TableChange.set("key-after-progress", "value")),
+                        false)
+                .get();
+
+        assertThatThrownBy(
+                        () ->
+                                admin.alterTable(
+                                                lakeEnabledTablePath,
+                                                Collections.singletonList(
+                                                        TableChange.set(
+                                                                ConfigOptions
+                                                                        .TABLE_DATALAKE_TABLE_NAME
+                                                                        .key(),
+                                                                "third_lake_table")),
+                                                false)
+                                        .get())
+                .cause()
+                .isInstanceOf(InvalidAlterTableException.class)
+                .hasMessageContaining(
+                        "cannot be altered when 'table.datalake.enabled' is true or the table has tiering progress")
+                .hasMessageContaining(ConfigOptions.TABLE_DATALAKE_TABLE_NAME.key());
+    }
+
+    @Test
+    void testAlterLakePathOptionsOnlyForPaimon() throws Exception {

Review Comment:
   This constraint belongs to create-time descriptor validation rather than ALTER behavior. Could we move this coverage to `TableDescriptorValidationTest` by validating a descriptor with a custom lake path against `DataLakeFormat.ICEBERG`, and remove this IT test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org