You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2026/10/17 21:07:35 UTC
(doris) branch master updated: [feature](catalog) Support altering Iceberg and Paimon table properties (#69086)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8b67d53f70b [feature](catalog) Support altering Iceberg and Paimon table properties (#69086)
8b67d53f70b is described below
commit 8b67d53f70b5f3c94d4ca23575eaa250cc3176ec
Author: Mingyu Chen (Rayner) <mo...@gmail.com>
AuthorDate: Sat Oct 17 22:07:23 2026 +0100
[feature](catalog) Support altering Iceberg and Paimon table properties (#69086)
### What problem does this PR solve?
Issue Number: None
Related PR: #66428
Problem Summary:
**In short.** This ports #66428 from branch-4.1 (it is also in
branch-4.2): `ALTER TABLE ... SET ("key" = "value", ...)` sets the
native properties of an Iceberg or Paimon table. On master the statement
fails analysis for any key other than `auto_analyze_policy`. This PR
adds a connector SPI method for it, implements it for Iceberg and
Paimon, and commits all the keys of one statement together or none of
them.
**Background.** On master every external catalog is plugin driven. An
`ALTER TABLE` on an external table reaches
`PluginDrivenExternalCatalog`, which resolves the connector's table
handle and calls the matching `ConnectorMetadata` method (column
evolution, branches and tags, partition evolution). The only `SET (...)`
an external table accepted was `auto_analyze_policy`, which Doris stores
itself. In branch-4.x, #66428 added `updateTableProperties` to the
Iceberg and Paimon metadata ops in fe-core. master has no such ops in
fe-core, and the SPI had no method for table properties.
**The problem, and what it cost.**
| Statement on an Iceberg or Paimon table | master | with this PR |
|---|---|---|
| `SET ('write.target-file-size-bytes' = '134217728')` | `Unknown table
property: [write.target-file-size-bytes]` | set on the table |
| `SET ('snapshot.num-retained.min' = '3', 'snapshot.num-retained.max' =
'6')` | `Can only set one table property(without dynamic partition &&
binlog) at a time` | both set, as one Paimon schema change |
| `SET ('auto_analyze_policy' = 'disable')` | stored by Doris |
unchanged |
| `SET ('auto_analyze_policy' = 'enable', 'write.target-file-size-bytes'
= '536870912')` | `Can only set one table property ...` |
`auto_analyze_policy cannot be set with external table properties` |
Properties such as snapshot retention, target file size or manifest
merging could only be changed from another engine, such as Spark or
Flink.
**How this PR fixes it.**
- SPI: `ConnectorTableDdlOps#setTableProperties(session, handle,
properties)`. The default throws `SET TABLE PROPERTIES not supported`. A
connector that implements it must commit every entry of the map or none
of them. The recorded method surface is updated.
- fe-core, as in branch-4.1: `ModifyTablePropertiesOp` accepts any
number of keys for a table outside the internal catalog and rejects
`auto_analyze_policy` mixed with other keys. `Alter` keeps
`auto_analyze_policy` on the Doris path and sends the other keys to
`CatalogIf#updateTableProperties`. `PluginDrivenExternalCatalog` calls
the connector, and writes the refresh edit log and refreshes the cached
table only after the connector returns.
- Iceberg: one `UpdateProperties` commit inside the catalog's auth
context. Iceberg validates the whole map before it commits, so a map
with a format-version downgrade does not commit its valid keys either.
- Paimon: one `Catalog.alterTable` with a `SchemaChange.setOption` per
key. Paimon validates the whole list before it commits, and one
statement adds one schema version.
- HMS: as it does for the other ALTER operations, the HMS gateway
forwards the call to its Iceberg sibling for an Iceberg table, so the
statement behaves the same through an HMS catalog. A Hive table still
reports it as unsupported. Branch-4.x does not support it through an HMS
catalog.
**Results.**
- Unit tests:
- `ModifyTablePropertiesOpTest`, from branch-4.1: validation for
external and internal tables, and of `auto_analyze_policy`.
- `PluginDrivenExternalCatalogDdlRoutingTest`: the whole map reaches the
connector in one call, and the refresh edit log and cache refresh
follow. A connector failure is reported, and neither of them happens.
- `CatalogBackedIcebergCatalogOpsDdlTest`, against an in-memory Iceberg
catalog: two keys add one metadata version, a later SET keeps the keys
it does not name, and a map with a downgrade commits nothing.
- `PaimonConnectorMetadataTablePropertiesTest`, against a local
filesystem catalog: two options add one schema version and keep the
options they do not name, and a list with an invalid field option
commits nothing.
- `IcebergConnectorMetadataDdlTest`,
`HiveConnectorMetadataSiblingDelegationTest` and
`ConnectorMetadataSurfaceTest` cover the auth wrapping, the HMS
forwarding and the SPI surface.
- Each of these changes turned the matching tests red: one alter per
Paimon option, one commit per Iceberg key, and refreshing before the
connector call. The full suites of fe-connector-spi, -hive, -iceberg and
-paimon pass.
- Regression, on a local cluster with the Iceberg docker environment
(REST catalog, MinIO, Spark):
- `test_iceberg_alter_properties` reads the properties back through
Spark after each statement.
- `test_paimon_alter_properties` checks `$options` and `$schemas`. Its
`.out` was generated there: the schema id goes 0, 1, 2 for the two SETs
and stays at 2 after the `auto_analyze_policy` statement and after the
rejected one.
- Both suites come from branch-4.1. The Paimon one records its results
Co-authored-by: Socrates <su...@selectdb.com>
---
.../connector/hive/HiveConnectorMetadata.java | 10 ++
...HiveConnectorMetadataSiblingDelegationTest.java | 13 +-
.../doris/connector/iceberg/IcebergCatalogOps.java | 17 +++
.../iceberg/IcebergConnectorMetadata.java | 21 +++
.../CatalogBackedIcebergCatalogOpsDdlTest.java | 58 +++++++++
.../iceberg/IcebergConnectorMetadataDdlTest.java | 30 +++++
.../iceberg/RecordingIcebergCatalogOps.java | 11 ++
.../connector/paimon/PaimonConnectorMetadata.java | 14 ++
...PaimonConnectorMetadataTablePropertiesTest.java | 142 +++++++++++++++++++++
.../doris/connector/spi/ConnectorTableDdlOps.java | 24 +++-
.../doris/connector/spi/ConnectorTableOps.java | 2 +-
.../test/resources/connector-metadata-methods.txt | 1 +
.../main/java/org/apache/doris/alter/Alter.java | 7 +-
.../org/apache/doris/datasource/CatalogIf.java | 6 +
.../apache/doris/datasource/ExternalCatalog.java | 7 +
.../plugin/PluginDrivenExternalCatalog.java | 21 +++
.../commands/info/ModifyTablePropertiesOp.java | 45 +++++--
.../PluginDrivenExternalCatalogDdlRoutingTest.java | 49 +++++++
.../commands/info/ModifyTablePropertiesOpTest.java | 99 ++++++++++++++
.../paimon/test_paimon_alter_properties.out | 30 +++++
.../iceberg/test_iceberg_alter_properties.groovy | 108 ++++++++++++++++
.../paimon/test_paimon_alter_properties.groovy | 122 ++++++++++++++++++
22 files changed, 821 insertions(+), 16 deletions(-)
diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
index 64e503c219f..6971d3763f5 100644
--- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
+++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
@@ -1947,6 +1947,16 @@ public class HiveConnectorMetadata implements ConnectorMetadata {
throw new DorisConnectorException("RENAME TABLE not supported");
}
+ @Override
+ public void setTableProperties(ConnectorSession session, ConnectorTableHandle handle,
+ Map<String, String> properties) {
+ if (!(handle instanceof HiveTableHandle)) {
+ siblingMetadata(session, handle).setTableProperties(session, handle, properties);
+ return;
+ }
+ throw new DorisConnectorException("SET TABLE PROPERTIES not supported");
+ }
+
@Override
public void addColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column,
ConnectorColumnPosition position) {
diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
index f086b010fc6..928fd132811 100644
--- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
+++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
@@ -363,6 +363,7 @@ public class HiveConnectorMetadataSiblingDelegationTest {
// run the hive branch and never be cast. Change objects are null — the guard fires on the handle type
// before any param is touched.
md.renameTable(session, foreignHandle, "new");
+ md.setTableProperties(session, foreignHandle, Collections.emptyMap());
md.addColumn(session, foreignHandle, null, null);
md.addColumns(session, foreignHandle, Collections.emptyList());
md.dropColumn(session, foreignHandle, "c");
@@ -420,6 +421,8 @@ public class HiveConnectorMetadataSiblingDelegationTest {
// Group-1: ALTER-DDL for a hive handle throws the EXACT inherited SPI-default message (byte-parity with
// pre-override behavior) without building or consulting the sibling.
assertThrowsMessage(() -> md.renameTable(session, hive, "n"), "RENAME TABLE not supported");
+ assertThrowsMessage(() -> md.setTableProperties(session, hive, Collections.emptyMap()),
+ "SET TABLE PROPERTIES not supported");
assertThrowsMessage(() -> md.addColumn(session, hive, null, null), "ADD COLUMN not supported");
assertThrowsMessage(() -> md.addColumns(session, hive, Collections.emptyList()), "ADD COLUMNS not supported");
assertThrowsMessage(() -> md.dropColumn(session, hive, "c"), "DROP COLUMN not supported");
@@ -739,8 +742,8 @@ public class HiveConnectorMetadataSiblingDelegationTest {
// The exact set + order of ALTER-DDL / validate methods the foreign-handle write test drives (Rule-9
// completeness lock for §4.4 W1: dropping a guard, or adding one that should not forward, fails the test).
static final List<String> EXPECTED_WRITE_METHODS = Collections.unmodifiableList(Arrays.asList(
- "renameTable", "addColumn", "addColumns", "dropColumn", "renameColumn", "modifyColumn",
- "reorderColumns", "addNestedColumn", "dropNestedColumn", "renameNestedColumn",
+ "renameTable", "setTableProperties", "addColumn", "addColumns", "dropColumn", "renameColumn",
+ "modifyColumn", "reorderColumns", "addNestedColumn", "dropNestedColumn", "renameNestedColumn",
"modifyNestedColumn", "modifyColumnComment",
"createOrReplaceBranch", "createOrReplaceTag", "dropBranch", "dropTag",
"addPartitionField", "dropPartitionField", "replacePartitionField",
@@ -941,6 +944,12 @@ public class HiveConnectorMetadataSiblingDelegationTest {
calls.add("renameTable");
}
+ @Override
+ public void setTableProperties(ConnectorSession session, ConnectorTableHandle handle,
+ Map<String, String> properties) {
+ calls.add("setTableProperties");
+ }
+
@Override
public void addColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column,
ConnectorColumnPosition position) {
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
index cb2d420ed6d..19776b91542 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
@@ -34,6 +34,7 @@ import org.apache.iceberg.SnapshotRef;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.Table;
import org.apache.iceberg.UpdatePartitionSpec;
+import org.apache.iceberg.UpdateProperties;
import org.apache.iceberg.UpdateSchema;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.Namespace;
@@ -139,6 +140,12 @@ public interface IcebergCatalogOps {
/** Renames {@code dbName.oldName} to {@code dbName.newName} (same database). */
void renameTable(String dbName, String oldName, String newName);
+ /**
+ * Sets {@code properties} on {@code dbName.tableName} in ONE {@code UpdateProperties} commit, so iceberg
+ * validates the whole map before any of it is written. Keys not in the map keep their current values.
+ */
+ void setTableProperties(String dbName, String tableName, Map<String, String> properties);
+
/** The table's storage location, or empty when blank — read BEFORE a drop to prune empty dirs. */
Optional<String> loadTableLocation(String dbName, String tableName);
@@ -476,6 +483,16 @@ public interface IcebergCatalogOps {
catalog.renameTable(toTableIdentifier(dbName, oldName), toTableIdentifier(dbName, newName));
}
+ @Override
+ public void setTableProperties(String dbName, String tableName, Map<String, String> properties) {
+ withTable(dbName, tableName, table -> {
+ UpdateProperties updateProperties = table.updateProperties();
+ properties.forEach(updateProperties::set);
+ updateProperties.commit();
+ return null;
+ });
+ }
+
@Override
public Optional<String> loadTableLocation(String dbName, String tableName) {
return withTable(dbName, tableName, table -> {
diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
index ec7cbd5cdda..0516fc0b15a 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
@@ -1239,6 +1239,27 @@ public class IcebergConnectorMetadata implements ConnectorMetadata {
}
}
+ /**
+ * Sets table properties for {@code ALTER TABLE ... SET (...)}: a thin seam delegation inside the auth
+ * context. The seam puts every entry into one {@code UpdateProperties} commit, so a map iceberg rejects
+ * (a format-version downgrade, say) leaves the table exactly as it was, valid entries included.
+ */
+ @Override
+ public void setTableProperties(ConnectorSession session, ConnectorTableHandle handle,
+ Map<String, String> properties) {
+ IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
+ try {
+ executeAuthenticated(() -> {
+ catalogOps.setTableProperties(iceHandle.getDbName(), iceHandle.getTableName(), properties);
+ return null;
+ });
+ } catch (Exception e) {
+ throw new DorisConnectorException("Failed to set properties for Iceberg table "
+ + iceHandle.getDbName() + "." + iceHandle.getTableName()
+ + ": " + ExceptionUtils.getRootCauseMessage(e), e);
+ }
+ }
+
// ========== Column evolution (B2) — mirror legacy IcebergMetadataOps add/drop/rename/modify/reorder ==========
/**
diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsDdlTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsDdlTest.java
index b2d75f202cb..75c1d25b4f1 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsDdlTest.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsDdlTest.java
@@ -29,12 +29,14 @@ import org.apache.doris.connector.spi.ddl.TagChange;
import org.apache.iceberg.DataFiles;
import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.HasTableOperations;
import org.apache.iceberg.PartitionField;
import org.apache.iceberg.PartitionSpec;
import org.apache.iceberg.Schema;
import org.apache.iceberg.SnapshotRef;
import org.apache.iceberg.SortOrder;
import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
import org.apache.iceberg.catalog.Namespace;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.inmemory.InMemoryCatalog;
@@ -46,6 +48,7 @@ import org.junit.jupiter.api.Test;
import java.util.Arrays;
import java.util.Collections;
+import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.atomic.AtomicInteger;
@@ -203,6 +206,61 @@ public class CatalogBackedIcebergCatalogOpsDdlTest {
Assertions.assertThrows(Exception.class, () -> ops.renameTable("db1", "ghost", "t2"));
}
+ // ---------- setTableProperties: one UpdateProperties commit on a real catalog ----------
+
+ @Test
+ public void testSetTablePropertiesCommitsWholeMapAsOneMetadataVersion() {
+ ops.createDatabase("db1", Collections.emptyMap());
+ ops.createTable("db1", "t1", schema(), PartitionSpec.unpartitioned(), null,
+ IcebergSchemaBuilder.buildTableProperties(Collections.emptyMap()));
+ int versions = metadataVersions("t1");
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES, "134217728");
+ properties.put(TableProperties.MANIFEST_MIN_MERGE_COUNT, "50");
+
+ ops.setTableProperties("db1", "t1", properties);
+
+ Map<String, String> actual = ops.loadTable("db1", "t1").properties();
+ Assertions.assertEquals("134217728", actual.get(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES));
+ Assertions.assertEquals("50", actual.get(TableProperties.MANIFEST_MIN_MERGE_COUNT));
+ Assertions.assertEquals(versions + 1, metadataVersions("t1"), "the whole map must be one commit");
+
+ // A later SET replaces the named property and keeps the others.
+ ops.setTableProperties("db1", "t1",
+ Collections.singletonMap(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES, "268435456"));
+ actual = ops.loadTable("db1", "t1").properties();
+ Assertions.assertEquals("268435456", actual.get(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES));
+ Assertions.assertEquals("50", actual.get(TableProperties.MANIFEST_MIN_MERGE_COUNT));
+ }
+
+ @Test
+ public void testSetTablePropertiesRejectedMapCommitsNothing() {
+ ops.createDatabase("db1", Collections.emptyMap());
+ ops.createTable("db1", "t1", schema(), PartitionSpec.unpartitioned(), null,
+ IcebergSchemaBuilder.buildTableProperties(Collections.emptyMap()));
+ int versions = metadataVersions("t1");
+ // The valid entry goes first: committing the entries one at a time would write it before the
+ // downgrade fails.
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES, "536870912");
+ properties.put(TableProperties.FORMAT_VERSION, "1");
+
+ IllegalArgumentException e = Assertions.assertThrows(IllegalArgumentException.class,
+ () -> ops.setTableProperties("db1", "t1", properties));
+
+ Assertions.assertTrue(e.getMessage().contains("Cannot downgrade"), e.getMessage());
+ Table table = ops.loadTable("db1", "t1");
+ Assertions.assertEquals(2, ((HasTableOperations) table).operations().current().formatVersion());
+ Assertions.assertFalse(table.properties().containsKey(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES),
+ "the valid entry of a rejected map must not be committed on its own");
+ Assertions.assertEquals(versions, metadataVersions("t1"));
+ }
+
+ private int metadataVersions(String tableName) {
+ Table table = ops.loadTable("db1", tableName);
+ return ((HasTableOperations) table).operations().current().previousFiles().size();
+ }
+
// ---------- Branch / tag (B4): real ManageSnapshots round-trips on an InMemoryCatalog ----------
/** Creates db1.t1 and seeds {@code snapshots} consecutive snapshots; returns the current snapshot id. */
diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataDdlTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataDdlTest.java
index 62c7eb3ea41..ff9dc79bc70 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataDdlTest.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataDdlTest.java
@@ -486,6 +486,36 @@ public class IcebergConnectorMetadataDdlTest {
Assertions.assertTrue(ops.log.isEmpty());
}
+ // ---------- setTableProperties ----------
+
+ @Test
+ public void testSetTablePropertiesRoutesWholeMapByHandleAndIsAuthWrapped() {
+ RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps();
+ RecordingConnectorContext ctx = new RecordingConnectorContext();
+ Map<String, String> properties = new HashMap<>();
+ properties.put(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES, "134217728");
+ properties.put(TableProperties.MANIFEST_MIN_MERGE_COUNT, "50");
+ metadata(ops, ctx, IcebergCatalogProperties.TYPE_REST)
+ .setTableProperties(null, new IcebergTableHandle("db1", "t1"), properties);
+ Assertions.assertEquals(1, ops.log.size(), ops.log.toString());
+ Assertions.assertEquals("db1", ops.lastSetPropertiesDb);
+ Assertions.assertEquals("t1", ops.lastSetPropertiesTable);
+ Assertions.assertEquals(properties, ops.lastSetProperties);
+ Assertions.assertEquals(1, ctx.authCount, "setTableProperties must run inside executeAuthenticated");
+ }
+
+ @Test
+ public void testSetTablePropertiesAuthFailureWraps() {
+ RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps();
+ RecordingConnectorContext ctx = new RecordingConnectorContext();
+ ctx.failAuth = true;
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> metadata(ops, ctx, IcebergCatalogProperties.TYPE_REST).setTableProperties(null,
+ new IcebergTableHandle("db1", "t1"),
+ Collections.singletonMap(TableProperties.WRITE_TARGET_FILE_SIZE_BYTES, "134217728")));
+ Assertions.assertTrue(ops.log.isEmpty());
+ }
+
// ---------- Branch / tag (B4): route by handle, auth-wrap, wrap auth failures ----------
diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingIcebergCatalogOps.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingIcebergCatalogOps.java
index 3ccbb91a571..d55aece7f25 100644
--- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingIcebergCatalogOps.java
+++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingIcebergCatalogOps.java
@@ -115,6 +115,9 @@ final class RecordingIcebergCatalogOps implements IcebergCatalogOps {
String lastRenameTableDb;
String lastRenameTableOld;
String lastRenameTableNew;
+ String lastSetPropertiesDb;
+ String lastSetPropertiesTable;
+ Map<String, String> lastSetProperties;
/** Canned location answers for the load-before-drop helpers (default: absent). */
Optional<String> tableLocation = Optional.empty();
Optional<String> namespaceLocation = Optional.empty();
@@ -282,6 +285,14 @@ final class RecordingIcebergCatalogOps implements IcebergCatalogOps {
lastRenameTableNew = newName;
}
+ @Override
+ public void setTableProperties(String dbName, String tableName, Map<String, String> properties) {
+ log.add("setTableProperties:" + dbName + "." + tableName + ":" + properties);
+ lastSetPropertiesDb = dbName;
+ lastSetPropertiesTable = tableName;
+ lastSetProperties = properties;
+ }
+
@Override
public Optional<String> loadTableLocation(String dbName, String tableName) {
log.add("loadTableLocation:" + dbName + "." + tableName);
diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java
index 2ff66cb9e8b..967d60df63a 100644
--- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java
+++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java
@@ -1193,6 +1193,20 @@ public class PaimonConnectorMetadata implements ConnectorMetadata {
LOG.info("dropped Paimon table {}", id);
}
+ /**
+ * Sets table options for {@code ALTER TABLE ... SET (...)}. Every entry becomes a
+ * {@link SchemaChange#setOption} in ONE {@code alterTable} call: paimon validates the whole list against
+ * the new schema before committing it, so a rejected entry keeps the valid ones from being written too,
+ * and the table gets one new schema version rather than one per option.
+ */
+ @Override
+ public void setTableProperties(ConnectorSession session, ConnectorTableHandle handle,
+ Map<String, String> properties) {
+ List<SchemaChange> changes = new ArrayList<>(properties.size());
+ properties.forEach((key, value) -> changes.add(SchemaChange.setOption(key, value)));
+ alterTable((PaimonTableHandle) handle, changes, "set properties");
+ }
+
// ==================== DDL: Column evolution ====================
@Override
diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTablePropertiesTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTablePropertiesTest.java
new file mode 100644
index 00000000000..c745ca9ee9a
--- /dev/null
+++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTablePropertiesTest.java
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.paimon;
+
+import org.apache.doris.connector.spi.DorisConnectorException;
+
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.FileSystemCatalog;
+import org.apache.paimon.catalog.Identifier;
+import org.apache.paimon.fs.local.LocalFileIO;
+import org.apache.paimon.schema.Schema;
+import org.apache.paimon.schema.SchemaChange;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.types.DataTypes;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * {@code ALTER TABLE ... SET (...)} on a Paimon table: every property becomes a {@link SchemaChange#setOption} in
+ * ONE remote {@code alterTable}, so paimon validates the whole list before committing any of it and the table gets
+ * one new schema version per statement.
+ */
+public class PaimonConnectorMetadataTablePropertiesTest {
+
+ private static final Identifier TABLE_ID = Identifier.create("db", "t");
+
+ @Test
+ public void setTablePropertiesSendsEveryOptionInOneAuthenticatedAlter() {
+ RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
+ RecordingConnectorContext context = new RecordingConnectorContext();
+ PaimonConnectorMetadata metadata = new PaimonConnectorMetadata(
+ ops, PaimonCatalogProperties.of(Collections.emptyMap()), context);
+
+ metadata.setTableProperties(null, handle(), options("2", "5"));
+
+ Assertions.assertEquals(Collections.singletonList("alterTable:db.t"), ops.log);
+ Assertions.assertEquals(TABLE_ID, ops.lastAlteredTableId);
+ Assertions.assertEquals(Arrays.asList(
+ SchemaChange.setOption("snapshot.num-retained.min", "2"),
+ SchemaChange.setOption("snapshot.num-retained.max", "5")),
+ ops.lastSchemaChanges);
+ Assertions.assertEquals(1, context.authCount, "the remote alter must run inside executeAuthenticated");
+ }
+
+ @Test
+ public void setTablePropertiesPersistsEveryOptionAsOneSchemaVersion(@TempDir Path warehouse) throws Exception {
+ try (Catalog catalog = localCatalog(warehouse)) {
+ catalog.createTable(TABLE_ID, Schema.newBuilder()
+ .column("id", DataTypes.INT())
+ .option("snapshot.num-retained.min", "1")
+ .option("snapshot.num-retained.max", "8")
+ .option("commit.max-retries", "3")
+ .build(), false);
+ long schemaId = schemaId(catalog);
+
+ metadataOver(catalog).setTableProperties(null, handle(), options("2", "5"));
+
+ Map<String, String> actual = catalog.getTable(TABLE_ID).options();
+ Assertions.assertEquals("2", actual.get("snapshot.num-retained.min"));
+ Assertions.assertEquals("5", actual.get("snapshot.num-retained.max"));
+ Assertions.assertEquals("3", actual.get("commit.max-retries"), "an option not named must be kept");
+ Assertions.assertEquals(schemaId + 1, schemaId(catalog), "one statement must be one schema version");
+ }
+ }
+
+ @Test
+ public void setTablePropertiesRejectedListCommitsNothing(@TempDir Path warehouse) throws Exception {
+ try (Catalog catalog = localCatalog(warehouse)) {
+ catalog.createTable(TABLE_ID, Schema.newBuilder()
+ .column("id", DataTypes.INT().notNull())
+ .column("seq", DataTypes.BIGINT())
+ .primaryKey("id")
+ .option("snapshot.num-retained.min", "2")
+ .option("snapshot.num-retained.max", "5")
+ .build(), false);
+ long schemaId = schemaId(catalog);
+ // The valid entry goes first: applying the entries one at a time would commit it before the
+ // invalid one fails.
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put("snapshot.num-retained.max", "10");
+ properties.put("fields.missing.sequence-group", "seq");
+
+ DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class,
+ () -> metadataOver(catalog).setTableProperties(null, handle(), properties));
+
+ Assertions.assertTrue(e.getMessage().contains("missing"), e.getMessage());
+ Map<String, String> actual = catalog.getTable(TABLE_ID).options();
+ Assertions.assertEquals("5", actual.get("snapshot.num-retained.max"),
+ "the valid entry of a rejected list must not be committed on its own");
+ Assertions.assertFalse(actual.containsKey("fields.missing.sequence-group"));
+ Assertions.assertEquals(schemaId, schemaId(catalog));
+ }
+ }
+
+ private static Map<String, String> options(String min, String max) {
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put("snapshot.num-retained.min", min);
+ properties.put("snapshot.num-retained.max", max);
+ return properties;
+ }
+
+ private static PaimonTableHandle handle() {
+ return new PaimonTableHandle("db", "t", Collections.emptyList(), Collections.emptyList());
+ }
+
+ private static Catalog localCatalog(Path warehouse) throws Exception {
+ Catalog catalog = new FileSystemCatalog(LocalFileIO.create(), new org.apache.paimon.fs.Path(warehouse.toUri()));
+ catalog.createDatabase("db", false);
+ return catalog;
+ }
+
+ private static PaimonConnectorMetadata metadataOver(Catalog catalog) {
+ return new PaimonConnectorMetadata(new PaimonCatalogOps.CatalogBackedPaimonCatalogOps(catalog),
+ PaimonCatalogProperties.of(Collections.emptyMap()), new RecordingConnectorContext());
+ }
+
+ private static long schemaId(Catalog catalog) throws Exception {
+ return ((FileStoreTable) catalog.getTable(TABLE_ID)).schema().id();
+ }
+}
diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableDdlOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableDdlOps.java
index 5fe962e7ec4..83923102b7d 100644
--- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableDdlOps.java
+++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableDdlOps.java
@@ -21,9 +21,10 @@ import org.apache.doris.connector.spi.ddl.ConnectorCreateTableRequest;
import org.apache.doris.connector.spi.handle.ConnectorTableHandle;
import java.util.List;
+import java.util.Map;
/**
- * Table-level DDL: create, drop, rename, truncate.
+ * Table-level DDL: create, drop, rename, truncate, set properties.
*
* <p><b>The whole domain is optional</b> — a read-only connector implements nothing here and every default
* fails loud with a "not supported" message, which is the correct answer for it.</p>
@@ -34,7 +35,8 @@ import java.util.List;
* degraded to it, silently dropping the partition spec, the bucket spec and {@code IF NOT EXISTS} — a
* connector implementing only the narrow form reported success on a partitioned {@code CREATE TABLE} and
* created an unpartitioned table. Nothing implemented it, so it is gone rather than documented.
- * {@link #dropTable}, {@link #renameTable} and {@link #truncateTable} are independent and optional.</p>
+ * {@link #dropTable}, {@link #renameTable}, {@link #truncateTable} and {@link #setTableProperties} are
+ * independent and optional.</p>
*/
public interface ConnectorTableDdlOps {
@@ -82,4 +84,22 @@ public interface ConnectorTableDdlOps {
throw new DorisConnectorException(
"TRUNCATE TABLE not supported");
}
+
+ /**
+ * Sets connector-native properties on the table identified by {@code handle}, for
+ * {@code ALTER TABLE ... SET ("key" = "value", ...)}. A key in {@code properties} is added or overwritten;
+ * a key not in it keeps its current value.
+ *
+ * <p>The map is one change: the connector must commit every entry or none of them. A statement that
+ * fails after committing part of the map would leave the table in a state the user never asked for.
+ * Properties Doris owns itself, such as {@code auto_analyze_policy}, never reach this method.</p>
+ *
+ * @throws DorisConnectorException if the connector does not support it, or the remote catalog rejects
+ * the change
+ */
+ default void setTableProperties(ConnectorSession session,
+ ConnectorTableHandle handle, Map<String, String> properties) {
+ throw new DorisConnectorException(
+ "SET TABLE PROPERTIES not supported");
+ }
}
diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableOps.java
index 41c4186fda5..609dbc28ddd 100644
--- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableOps.java
+++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableOps.java
@@ -28,7 +28,7 @@ package org.apache.doris.connector.spi;
* <li>{@link ConnectorTableMetadataOps} — resolving a table by name and reading what it looks like. The
* one domain no connector can skip.</li>
* <li>{@link ConnectorViewOps} — views.</li>
- * <li>{@link ConnectorTableDdlOps} — create / drop / rename / truncate.</li>
+ * <li>{@link ConnectorTableDdlOps} — create / drop / rename / truncate / set properties.</li>
* <li>{@link ConnectorColumnEvolutionOps} — column schema evolution, flat and nested.</li>
* <li>{@link ConnectorSnapshotRefOps} — branches, tags and partition-spec evolution.</li>
* <li>{@link ConnectorPartitionListingOps} — enumerating partitions.</li>
diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt
index b69475f1997..80057cd9be8 100644
--- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt
+++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt
@@ -65,6 +65,7 @@ renderShowCreateTableDdl(org.apache.doris.connector.spi.ConnectorSession,org.apa
reorderColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List)
replacePartitionField(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.ddl.PartitionFieldChange)
resolveTimeTravel(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec)
+setTableProperties(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Map)
supportsCastPredicatePushdown(org.apache.doris.connector.spi.ConnectorSession)
supportsColumnHandleSnapshotPin(org.apache.doris.connector.spi.ConnectorSession)
truncateTable(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
index 02afd90fc85..d64e5a94a07 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/Alter.java
@@ -446,7 +446,12 @@ public class Alter {
ExternalTable table, List<AlterOp> alterOps) throws UserException {
for (AlterOp alterOp : alterOps) {
if (alterOp instanceof ModifyTablePropertiesOp) {
- setExternalTableAutoAnalyzePolicy(table, alterOps);
+ Map<String, String> properties = alterOp.getProperties();
+ if (properties.containsKey(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY)) {
+ setExternalTableAutoAnalyzePolicy(table, alterOps);
+ } else {
+ table.getCatalog().updateTableProperties(table, properties);
+ }
} else if (alterOp instanceof CreateOrReplaceBranchOp) {
table.getCatalog().createOrReplaceBranch(
table, ((CreateOrReplaceBranchOp) alterOp).getBranchInfo());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
index 13260a4a912..584a4b1b2d5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
@@ -362,6 +362,12 @@ public interface CatalogIf<T extends DatabaseIf> {
throw new UserException("Not support reorder columns operation");
}
+ // ALTER TABLE ... SET (...) with connector-native properties; auto_analyze_policy is handled by Doris itself
+
+ default void updateTableProperties(TableIf table, Map<String, String> properties) throws UserException {
+ throw new UserException("Not support update table properties operation");
+ }
+
// partition evolution operations (Iceberg): add / drop / replace a partition field
default void addPartitionField(TableIf table, AddPartitionFieldOp op) throws UserException {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index 9073222dab8..917d67a8de8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -1863,4 +1863,11 @@ public abstract class ExternalCatalog
Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName());
throw new DdlException("Reorder columns operation is not supported for catalog: " + getName());
}
+
+ @Override
+ public void updateTableProperties(TableIf dorisTable, Map<String, String> properties) throws UserException {
+ makeSureInitialized();
+ Preconditions.checkState(dorisTable instanceof ExternalTable, dorisTable.getName());
+ throw new DdlException("Update table properties operation is not supported for catalog: " + getName());
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
index 11e13d25969..60c2a5422e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java
@@ -1390,6 +1390,27 @@ public class PluginDrivenExternalCatalog extends ExternalCatalog {
afterExternalDdl(externalTable, updateTime);
}
+ /**
+ * Routes {@code ALTER TABLE ... SET ("key" = "value", ...)} with connector-native properties through the
+ * SPI's {@code ConnectorTableOps#setTableProperties}. The whole map goes to the connector in one call, which
+ * commits all of it or none of it, and {@link #afterExternalDdl} runs only after that call returns, so a
+ * rejected update neither writes the refresh editlog nor refreshes the cached table.
+ */
+ @Override
+ public void updateTableProperties(TableIf dorisTable, Map<String, String> properties) throws UserException {
+ ExternalTable externalTable = checkExternalTable(dorisTable);
+ ConnectorSession session = buildConnectorSession();
+ ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector);
+ ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata);
+ long updateTime = System.currentTimeMillis();
+ try {
+ metadata.setTableProperties(session, handle, properties);
+ } catch (DorisConnectorException e) {
+ throw new DdlException(e.getMessage(), e);
+ }
+ afterExternalDdl(externalTable, updateTime);
+ }
+
/** Initializes + checks the table is an {@link ExternalTable}, mirroring the base {@link ExternalCatalog}. */
private ExternalTable checkExternalTable(TableIf dorisTable) {
makeSureInitialized();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOp.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOp.java
index 86305e555ef..3e446c539e6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOp.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOp.java
@@ -95,6 +95,11 @@ public class ModifyTablePropertiesOp extends AlterTableOp {
throw new AnalysisException("Properties is not set");
}
+ if (tableName != null && !InternalCatalog.INTERNAL_CATALOG_NAME.equals(tableName.getCtl())) {
+ validateExternalTableProperties();
+ return;
+ }
+
if (properties.size() != 1
&& !TableProperty.isSamePrefixProperties(
properties, DynamicPartitionProperty.DYNAMIC_PARTITION_PROPERTY_PREFIX)
@@ -359,16 +364,7 @@ public class ModifyTablePropertiesOp extends AlterTableOp {
} else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_ROW_STORE_COLUMNS)) {
// do nothing, will be analyzed when creating alter job
} else if (properties.containsKey(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY)) {
- String analyzePolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY, "");
- if (analyzePolicy != null
- && !analyzePolicy.equals(PropertyAnalyzer.ENABLE_AUTO_ANALYZE_POLICY)
- && !analyzePolicy.equals(PropertyAnalyzer.DISABLE_AUTO_ANALYZE_POLICY)
- && !analyzePolicy.equals(PropertyAnalyzer.USE_CATALOG_AUTO_ANALYZE_POLICY)) {
- throw new AnalysisException(
- "Table auto analyze policy only support for " + PropertyAnalyzer.ENABLE_AUTO_ANALYZE_POLICY
- + " or " + PropertyAnalyzer.DISABLE_AUTO_ANALYZE_POLICY
- + " or " + PropertyAnalyzer.USE_CATALOG_AUTO_ANALYZE_POLICY);
- }
+ validateAutoAnalyzePolicy();
this.opType = AlterOpType.MODIFY_TABLE_PROPERTY_SYNC;
} else if (properties.containsKey(PropertyAnalyzer.ENABLE_UNIQUE_KEY_SKIP_BITMAP_COLUMN)) {
throw new AnalysisException("You can not modify property 'enable_unique_key_skip_bitmap_column'.");
@@ -383,6 +379,35 @@ public class ModifyTablePropertiesOp extends AlterTableOp {
analyzeForMTMV();
}
+ /**
+ * An external table takes its properties as connector-native options, any number of them in one
+ * statement, except {@code auto_analyze_policy}: Doris owns that one and stores it itself, so it must be
+ * set on its own rather than travel to the remote catalog with the others.
+ */
+ private void validateExternalTableProperties() throws AnalysisException {
+ this.opType = AlterOpType.MODIFY_TABLE_PROPERTY_SYNC;
+ if (!properties.containsKey(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY)) {
+ return;
+ }
+ if (properties.size() != 1) {
+ throw new AnalysisException("auto_analyze_policy cannot be set with external table properties");
+ }
+ validateAutoAnalyzePolicy();
+ }
+
+ private void validateAutoAnalyzePolicy() throws AnalysisException {
+ String analyzePolicy = properties.getOrDefault(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY, "");
+ if (analyzePolicy != null
+ && !analyzePolicy.equals(PropertyAnalyzer.ENABLE_AUTO_ANALYZE_POLICY)
+ && !analyzePolicy.equals(PropertyAnalyzer.DISABLE_AUTO_ANALYZE_POLICY)
+ && !analyzePolicy.equals(PropertyAnalyzer.USE_CATALOG_AUTO_ANALYZE_POLICY)) {
+ throw new AnalysisException(
+ "Table auto analyze policy only support for " + PropertyAnalyzer.ENABLE_AUTO_ANALYZE_POLICY
+ + " or " + PropertyAnalyzer.DISABLE_AUTO_ANALYZE_POLICY
+ + " or " + PropertyAnalyzer.USE_CATALOG_AUTO_ANALYZE_POLICY);
+ }
+ }
+
private void analyzeForMTMV() throws AnalysisException {
if (tableName != null) {
// Skip external catalog.
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java
index 57900ae241d..9605674d694 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java
@@ -1514,6 +1514,55 @@ public class PluginDrivenExternalCatalogDdlRoutingTest {
.addPartitionField(Mockito.any(), Mockito.any(), Mockito.any());
}
+ // ==================== SET TABLE PROPERTIES ====================
+
+ @Test
+ public void testUpdateTablePropertiesRoutesWholeMapAndRefreshes() throws Exception {
+ ExternalTable table = mockAlterTable();
+ ConnectorTableHandle handle = stubAlterHandle();
+ Map<String, String> properties = new HashMap<>();
+ properties.put("snapshot.num-retained.min", "2");
+ properties.put("snapshot.num-retained.max", "5");
+
+ catalog.updateTableProperties(table, properties);
+
+ // One connector call carries every property, so the connector can commit them as one change.
+ Mockito.verify(metadata).setTableProperties(session, handle, properties);
+ ArgumentCaptor<ExternalObjectLog> logCap = ArgumentCaptor.forClass(ExternalObjectLog.class);
+ Mockito.verify(mockEditLog).logRefreshExternalTable(logCap.capture());
+ Assertions.assertEquals("db1", logCap.getValue().getDbName());
+ Assertions.assertEquals("t1", logCap.getValue().getTableName());
+ Mockito.verify(mockRefreshManager).refreshTableInternal(
+ Mockito.eq(table.getDb()), Mockito.eq(table), Mockito.anyLong());
+ }
+
+ @Test
+ public void testUpdateTablePropertiesFailureSkipsRefresh() {
+ ExternalTable table = mockAlterTable();
+ ConnectorTableHandle handle = stubAlterHandle();
+ Mockito.doThrow(new DorisConnectorException("Cannot downgrade v2 table to v1"))
+ .when(metadata).setTableProperties(Mockito.eq(session), Mockito.eq(handle), Mockito.any());
+
+ DdlException ex = Assertions.assertThrows(DdlException.class, () -> catalog.updateTableProperties(
+ table, Collections.singletonMap("format-version", "1")));
+ Assertions.assertTrue(ex.getMessage().contains("Cannot downgrade"));
+ // A rejected update changed nothing remotely: no refresh editlog, no cache refresh.
+ Mockito.verify(mockEditLog, Mockito.never()).logRefreshExternalTable(Mockito.any());
+ Mockito.verify(mockRefreshManager, Mockito.never())
+ .refreshTableInternal(Mockito.any(), Mockito.any(), Mockito.anyLong());
+ }
+
+ @Test
+ public void testUpdateTablePropertiesHandleAbsentThrows() {
+ ExternalTable table = mockAlterTable();
+ Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")).thenReturn(Optional.empty());
+
+ Assertions.assertThrows(DdlException.class, () -> catalog.updateTableProperties(
+ table, Collections.singletonMap("snapshot.num-retained.max", "5")));
+ Mockito.verify(metadata, Mockito.never())
+ .setTableProperties(Mockito.any(), Mockito.any(), Mockito.any());
+ }
+
// ==================== helpers ====================
/** A mock external table whose LOCAL names are db1.t1 and REMOTE names DB1.TBL1 (name mapping enabled). */
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOpTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOpTest.java
new file mode 100644
index 00000000000..8e951e12207
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOpTest.java
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands.info;
+
+import org.apache.doris.alter.AlterOpType;
+import org.apache.doris.catalog.info.TableNameInfo;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.qe.ConnectContext;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ModifyTablePropertiesOpTest {
+
+ @Test
+ public void testExternalTablePropertiesAllowMultipleConnectorOptions() throws Exception {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("snapshot.num-retained.min", "2");
+ properties.put("snapshot.num-retained.max", "5");
+ ModifyTablePropertiesOp op = new ModifyTablePropertiesOp(properties);
+ op.setTableName(new TableNameInfo("paimon", "db", "tbl"));
+
+ op.validate(new ConnectContext());
+
+ Assertions.assertEquals(AlterOpType.MODIFY_TABLE_PROPERTY_SYNC, op.getOpType());
+ }
+
+ @Test
+ public void testInternalTableStillRejectsMultipleUnrelatedProperties() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("snapshot.num-retained.min", "2");
+ properties.put("snapshot.num-retained.max", "5");
+ ModifyTablePropertiesOp op = new ModifyTablePropertiesOp(properties);
+ op.setTableName(new TableNameInfo(InternalCatalog.INTERNAL_CATALOG_NAME, "db", "tbl"));
+
+ AnalysisException exception = Assertions.assertThrows(
+ AnalysisException.class, () -> op.validate(new ConnectContext()));
+ Assertions.assertTrue(exception.getMessage().contains("Can only set one table property"));
+ }
+
+ @Test
+ public void testExternalAutoAnalyzePolicyCannotBeMixedWithConnectorOptions() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY,
+ PropertyAnalyzer.ENABLE_AUTO_ANALYZE_POLICY);
+ properties.put("write.target-file-size-bytes", "134217728");
+ ModifyTablePropertiesOp op = new ModifyTablePropertiesOp(properties);
+ op.setTableName(new TableNameInfo("iceberg", "db", "tbl"));
+
+ AnalysisException exception = Assertions.assertThrows(
+ AnalysisException.class, () -> op.validate(new ConnectContext()));
+ Assertions.assertTrue(exception.getMessage().contains("cannot be set with external table properties"));
+ }
+
+ @Test
+ public void testExternalAutoAnalyzePolicyIsValidated() {
+ ModifyTablePropertiesOp op = new ModifyTablePropertiesOp(
+ Collections.singletonMap(PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY, "invalid"));
+ op.setTableName(new TableNameInfo("iceberg", "db", "tbl"));
+
+ AnalysisException exception = Assertions.assertThrows(
+ AnalysisException.class, () -> op.validate(new ConnectContext()));
+ Assertions.assertTrue(exception.getMessage().contains("Table auto analyze policy only support"));
+ }
+
+ @Test
+ public void testExternalAutoAnalyzePolicyKeepsDorisPropertyPath() throws Exception {
+ ModifyTablePropertiesOp op = new ModifyTablePropertiesOp(
+ Collections.singletonMap(
+ PropertyAnalyzer.PROPERTIES_AUTO_ANALYZE_POLICY,
+ PropertyAnalyzer.DISABLE_AUTO_ANALYZE_POLICY));
+ op.setTableName(new TableNameInfo("paimon", "db", "tbl"));
+
+ op.validate(new ConnectContext());
+
+ Assertions.assertEquals(AlterOpType.MODIFY_TABLE_PROPERTY_SYNC, op.getOpType());
+ }
+}
diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_alter_properties.out b/regression-test/data/external_table_p0/paimon/test_paimon_alter_properties.out
new file mode 100644
index 00000000000..ba7bb871388
--- /dev/null
+++ b/regression-test/data/external_table_p0/paimon/test_paimon_alter_properties.out
@@ -0,0 +1,30 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !created_schema --
+0
+
+-- !multi_set_schema --
+1
+
+-- !multi_set_options --
+snapshot.num-retained.max 6
+snapshot.num-retained.min 3
+
+-- !single_set_schema --
+2
+
+-- !single_set_options --
+snapshot.num-retained.max 8
+snapshot.num-retained.min 3
+
+-- !doris_owned_schema --
+2
+
+-- !doris_owned_options --
+snapshot.num-retained.max 8
+
+-- !rejected_schema --
+2
+
+-- !rejected_options --
+snapshot.num-retained.max 8
+
diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_alter_properties.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_alter_properties.groovy
new file mode 100644
index 00000000000..ebda448eba3
--- /dev/null
+++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_alter_properties.groovy
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_iceberg_alter_properties", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("Iceberg test is not enabled, skip this test")
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String catalogName = "test_iceberg_alter_properties"
+ String dbName = "test_iceberg_alter_properties_db"
+
+ // The properties are read back through Spark: what Doris commits must be
+ // what another engine sees in the Iceberg metadata.
+ def sparkProperty = { String key ->
+ spark_iceberg """REFRESH TABLE demo.${dbName}.iceberg_alter_properties"""
+ List<List<Object>> rows = spark_iceberg """
+ SHOW TBLPROPERTIES demo.${dbName}.iceberg_alter_properties ('${key}')
+ """
+ assertEquals(1, rows.size())
+ return rows[0][1].toString()
+ }
+
+ sql """DROP CATALOG IF EXISTS ${catalogName}"""
+ sql """
+ CREATE CATALOG ${catalogName} PROPERTIES (
+ 'type' = 'iceberg',
+ 'iceberg.catalog.type' = 'rest',
+ 'uri' = 'http://${externalEnvIp}:${restPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.region' = 'us-east-1'
+ )
+ """
+ sql """SWITCH ${catalogName}"""
+ sql """DROP DATABASE IF EXISTS ${dbName} FORCE"""
+ sql """CREATE DATABASE ${dbName}"""
+ sql """USE ${dbName}"""
+
+ sql """DROP TABLE IF EXISTS iceberg_alter_properties"""
+ sql """CREATE TABLE iceberg_alter_properties (id INT) PROPERTIES ('format-version' = '2')"""
+
+ // Multiple connector properties are committed together.
+ sql """
+ ALTER TABLE iceberg_alter_properties SET (
+ 'write.target-file-size-bytes' = '134217728',
+ 'commit.manifest.min-count-to-merge' = '50'
+ )
+ """
+ assertEquals("134217728", sparkProperty("write.target-file-size-bytes"))
+ assertEquals("50", sparkProperty("commit.manifest.min-count-to-merge"))
+
+ // A later SET replaces the selected property and preserves unrelated ones.
+ sql """
+ ALTER TABLE iceberg_alter_properties SET (
+ 'write.target-file-size-bytes' = '268435456'
+ )
+ """
+ assertEquals("268435456", sparkProperty("write.target-file-size-bytes"))
+ assertEquals("50", sparkProperty("commit.manifest.min-count-to-merge"))
+
+ // Doris-owned properties stay on the Doris path and never reach Iceberg.
+ sql """ALTER TABLE iceberg_alter_properties SET ('auto_analyze_policy' = 'disable')"""
+ assertEquals("268435456", sparkProperty("write.target-file-size-bytes"))
+ test {
+ sql """
+ ALTER TABLE iceberg_alter_properties SET (
+ 'auto_analyze_policy' = 'enable',
+ 'write.target-file-size-bytes' = '536870912'
+ )
+ """
+ exception "auto_analyze_policy cannot be set with external table properties"
+ }
+ assertEquals("268435456", sparkProperty("write.target-file-size-bytes"))
+
+ // Iceberg validates the whole UpdateProperties commit. A rejected format
+ // downgrade must not commit the otherwise valid size change.
+ test {
+ sql """
+ ALTER TABLE iceberg_alter_properties SET (
+ 'format-version' = '1',
+ 'write.target-file-size-bytes' = '536870912'
+ )
+ """
+ exception "Cannot downgrade"
+ }
+ assertEquals("2", sparkProperty("format-version"))
+ assertEquals("268435456", sparkProperty("write.target-file-size-bytes"))
+}
diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_alter_properties.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_alter_properties.groovy
new file mode 100644
index 00000000000..d504d89a0f6
--- /dev/null
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_alter_properties.groovy
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_paimon_alter_properties", "p0,external,paimon") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable paimon test.")
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String catalogName = "test_paimon_alter_properties"
+ String dbName = "test_paimon_alter_properties_db"
+
+ sql """DROP CATALOG IF EXISTS `${catalogName}`"""
+ sql """
+ CREATE CATALOG `${catalogName}` PROPERTIES (
+ 'type' = 'paimon',
+ 'paimon.catalog.type' = 'filesystem',
+ 'warehouse' = 's3://warehouse/wh',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minioPort}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.path.style.access' = 'true'
+ )
+ """
+ sql """SWITCH `${catalogName}`"""
+ sql """DROP DATABASE IF EXISTS `${dbName}` FORCE"""
+ sql """CREATE DATABASE `${dbName}`"""
+ sql """USE `${dbName}`"""
+ sql """DROP TABLE IF EXISTS `paimon_alter_properties`"""
+ sql """
+ CREATE TABLE `paimon_alter_properties` (
+ id INT NOT NULL,
+ seq BIGINT NULL,
+ payload STRING NULL
+ ) ENGINE=paimon
+ PROPERTIES (
+ 'primary-key' = 'id',
+ 'snapshot.num-retained.min' = '2',
+ 'snapshot.num-retained.max' = '5'
+ )
+ """
+ order_qt_created_schema """SELECT MAX(schema_id) FROM `paimon_alter_properties\$schemas`"""
+
+ // One Doris statement becomes one atomic Paimon schema change containing
+ // every SetOption, so the schema id moves by one. The refreshed system
+ // tables are visible immediately.
+ sql """
+ ALTER TABLE `paimon_alter_properties` SET (
+ 'snapshot.num-retained.min' = '3',
+ 'snapshot.num-retained.max' = '6'
+ )
+ """
+ order_qt_multi_set_schema """SELECT MAX(schema_id) FROM `paimon_alter_properties\$schemas`"""
+ order_qt_multi_set_options """
+ SELECT `key`, value FROM `paimon_alter_properties\$options`
+ WHERE `key` IN ('snapshot.num-retained.min', 'snapshot.num-retained.max')
+ """
+
+ // Updating one option replaces it without removing the other option.
+ sql """
+ ALTER TABLE `paimon_alter_properties` SET (
+ 'snapshot.num-retained.max' = '8'
+ )
+ """
+ order_qt_single_set_schema """SELECT MAX(schema_id) FROM `paimon_alter_properties\$schemas`"""
+ order_qt_single_set_options """
+ SELECT `key`, value FROM `paimon_alter_properties\$options`
+ WHERE `key` IN ('snapshot.num-retained.min', 'snapshot.num-retained.max')
+ """
+
+ // auto_analyze_policy is Doris metadata, so it does not create a Paimon
+ // schema version. Mixing the two property domains is rejected up front.
+ sql """ALTER TABLE `paimon_alter_properties` SET ('auto_analyze_policy' = 'disable')"""
+ test {
+ sql """
+ ALTER TABLE `paimon_alter_properties` SET (
+ 'auto_analyze_policy' = 'enable',
+ 'snapshot.num-retained.max' = '9'
+ )
+ """
+ exception "auto_analyze_policy cannot be set with external table properties"
+ }
+ order_qt_doris_owned_schema """SELECT MAX(schema_id) FROM `paimon_alter_properties\$schemas`"""
+ order_qt_doris_owned_options """
+ SELECT `key`, value FROM `paimon_alter_properties\$options`
+ WHERE `key` IN ('snapshot.num-retained.max', 'auto_analyze_policy')
+ """
+
+ // Paimon validates all SetOption changes before committing the next schema.
+ // A bad field-scoped option must not leak the valid max-retention update.
+ test {
+ sql """
+ ALTER TABLE `paimon_alter_properties` SET (
+ 'snapshot.num-retained.max' = '10',
+ 'fields.missing.sequence-group' = 'seq'
+ )
+ """
+ exception "missing"
+ }
+ order_qt_rejected_schema """SELECT MAX(schema_id) FROM `paimon_alter_properties\$schemas`"""
+ order_qt_rejected_options """
+ SELECT `key`, value FROM `paimon_alter_properties\$options`
+ WHERE `key` IN ('snapshot.num-retained.max', 'fields.missing.sequence-group')
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org