You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "johntomcat7408-cmyk (via GitHub)" <gi...@apache.org> on 2026/07/21 16:17:19 UTC
[PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
johntomcat7408-cmyk opened a new pull request, #4284:
URL: https://github.com/apache/amoro/pull/4284
## Summary
This is the second PR for #4240.
- Honor `CreateTableRequest.stageCreate()` for internal Iceberg REST catalogs.
- Return staged metadata without writing files to the table location or registering the table in AMS.
- Complete table creation when the client sends the following create commit.
- Keep direct table creation and normal metadata commits unchanged.
- Add an integration test for the staged-create transaction.
## Why
Trino expects the target table location to remain empty between staged creation and the final commit. AMS previously wrote the initial metadata file during the staged request, so Trino rejected the location as non-empty.
## Verification
- `mvn -ntp validate -pl amoro-ams -am -DskipTests -Pskip-dashboard-build`
- `mvn -ntp test -pl amoro-ams -am
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
Posted by "johntomcat7408-cmyk (via GitHub)" <gi...@apache.org>.
johntomcat7408-cmyk commented on PR #4284:
URL: https://github.com/apache/amoro/pull/4284#issuecomment-5129298520
@zhoujinsong Thanks for rerunning the workflow. Attempt 2 passed all four Hadoop 3 matrix jobs, including JDK 17 / Spark 3.5. I updated #4303 with the result; it will track the intermittent test-timing issue separately and should not block this PR. All required CI checks on the current head are now green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #4284:
URL: https://github.com/apache/amoro/pull/4284#discussion_r3673868395
##########
amoro-ams/src/main/java/org/apache/amoro/server/table/internal/InternalTableCreator.java:
##########
@@ -25,16 +25,19 @@
/** Interface to create an internal table. */
public interface InternalTableCreator extends Closeable {
- /**
- * Do all things about create an internal table, and prepare the {@link TableMetadata} for {@link
- * InternalTableManager#createTable(java.lang.String, TableMetadata)}
- */
+ /** Build Iceberg metadata without writing files or registering the table. */
+ org.apache.iceberg.TableMetadata stage();
Review Comment:
It seems the FQCN here is not necessary.
##########
amoro-ams/src/main/java/org/apache/amoro/server/RestCatalogService.java:
##########
@@ -358,25 +366,101 @@ public void loadTable(Context ctx) {
/** POST PREFIX/v1/catalogs/{catalog}/namespaces/{namespace}/tables/{table} */
public void commitTable(Context ctx) {
- handleTable(
- ctx,
- handler -> {
- UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
- TableOperations ops = handler.newTableOperator();
- TableMetadata base = ops.current();
- if (base == null) {
- throw new CommitFailedException("table metadata lost.");
- }
+ UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
+ request.validate();
+ if (isCreate(request)) {
+ handleNamespace(
+ ctx,
+ (catalog, database) -> {
+ String tableName = ctx.pathParam("table");
+ Preconditions.checkNotNull(tableName, "table name is null");
+ return commitCreateTable(catalog, database, tableName, request);
+ });
+ } else {
+ handleTable(
+ ctx,
+ handler -> {
+ TableOperations ops = handler.newTableOperator();
+ TableMetadata base = ops.current();
+ if (base == null) {
+ throw new CommitFailedException("table metadata lost.");
+ }
- TableMetadata.Builder builder = TableMetadata.buildFrom(base);
- request.requirements().forEach(r -> r.validate(base));
- request.updates().forEach(u -> u.applyTo(builder));
- TableMetadata newMetadata = builder.build();
+ TableMetadata.Builder builder = TableMetadata.buildFrom(base);
+ request.requirements().forEach(r -> r.validate(base));
+ request.updates().forEach(u -> u.applyTo(builder));
+ TableMetadata newMetadata = builder.build();
- ops.commit(base, newMetadata);
- TableMetadata current = ops.current();
- return LoadTableResponse.builder().withTableMetadata(current).build();
- });
+ ops.commit(base, newMetadata);
+ TableMetadata current = ops.current();
+ return LoadTableResponse.builder().withTableMetadata(current).build();
+ });
+ }
+ }
+
+ private LoadTableResponse commitCreateTable(
+ InternalCatalog catalog, String database, String tableName, UpdateTableRequest request) {
+ request.requirements().forEach(requirement -> requirement.validate((TableMetadata) null));
+
+ Optional<Integer> formatVersion =
+ request.updates().stream()
+ .filter(MetadataUpdate.UpgradeFormatVersion.class::isInstance)
+ .map(MetadataUpdate.UpgradeFormatVersion.class::cast)
+ .map(MetadataUpdate.UpgradeFormatVersion::formatVersion)
+ .findFirst();
+ TableMetadata.Builder builder =
Review Comment:
Format version from `UpgradeFormatVersion` is not applied to the builder,
You may change the format version like:
```java
TableMetadata.Builder builder = TableMetadata.buildFromEmpty();
formatVersion.ifPresent(builder::upgradeFormatVersion);
```
##########
amoro-ams/src/main/java/org/apache/amoro/server/RestCatalogService.java:
##########
@@ -358,25 +366,101 @@ public void loadTable(Context ctx) {
/** POST PREFIX/v1/catalogs/{catalog}/namespaces/{namespace}/tables/{table} */
public void commitTable(Context ctx) {
- handleTable(
- ctx,
- handler -> {
- UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
- TableOperations ops = handler.newTableOperator();
- TableMetadata base = ops.current();
- if (base == null) {
- throw new CommitFailedException("table metadata lost.");
- }
+ UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
+ request.validate();
+ if (isCreate(request)) {
+ handleNamespace(
+ ctx,
+ (catalog, database) -> {
+ String tableName = ctx.pathParam("table");
+ Preconditions.checkNotNull(tableName, "table name is null");
+ return commitCreateTable(catalog, database, tableName, request);
+ });
+ } else {
+ handleTable(
+ ctx,
+ handler -> {
+ TableOperations ops = handler.newTableOperator();
+ TableMetadata base = ops.current();
+ if (base == null) {
+ throw new CommitFailedException("table metadata lost.");
+ }
- TableMetadata.Builder builder = TableMetadata.buildFrom(base);
- request.requirements().forEach(r -> r.validate(base));
- request.updates().forEach(u -> u.applyTo(builder));
- TableMetadata newMetadata = builder.build();
+ TableMetadata.Builder builder = TableMetadata.buildFrom(base);
+ request.requirements().forEach(r -> r.validate(base));
+ request.updates().forEach(u -> u.applyTo(builder));
+ TableMetadata newMetadata = builder.build();
- ops.commit(base, newMetadata);
- TableMetadata current = ops.current();
- return LoadTableResponse.builder().withTableMetadata(current).build();
- });
+ ops.commit(base, newMetadata);
+ TableMetadata current = ops.current();
+ return LoadTableResponse.builder().withTableMetadata(current).build();
+ });
+ }
+ }
+
+ private LoadTableResponse commitCreateTable(
+ InternalCatalog catalog, String database, String tableName, UpdateTableRequest request) {
+ request.requirements().forEach(requirement -> requirement.validate((TableMetadata) null));
+
+ Optional<Integer> formatVersion =
+ request.updates().stream()
+ .filter(MetadataUpdate.UpgradeFormatVersion.class::isInstance)
+ .map(MetadataUpdate.UpgradeFormatVersion.class::cast)
+ .map(MetadataUpdate.UpgradeFormatVersion::formatVersion)
+ .findFirst();
+ TableMetadata.Builder builder =
+ formatVersion.map(TableMetadata::buildFromEmpty).orElseGet(TableMetadata::buildFromEmpty);
+ request.updates().forEach(update -> update.applyTo(builder));
+ TableMetadata icebergMetadata = builder.build();
+
+ CreateTableRequest createRequest =
Review Comment:
The `icebergMetadata` built from `UpdateTableRequest` updates already contains everything needed (schema, spec, sort order, location, properties, and appended data files).
The reconstructed `CreateTableRequest` is only used to construct the `InternalTableCreator` object, but `creator.create(icebergMetadata)` writes the `icebergMetadata` directly — the `CreateTableRequest` fields are not used for the actual metadata file.
This works correctly today but is confusing to read. Could you add a comment explaining that `CreateTableRequest` here is only to satisfy the current `InternalTableCreator` constructor signature, and the actual metadata written is `icebergMetadata`?
Maybe we can refactor the `InternalTableCreator` later.
##########
amoro-ams/src/test/java/org/apache/amoro/server/TestInternalIcebergCatalogService.java:
##########
@@ -284,6 +289,29 @@ public void testCreateTableInNamespaceLocation() {
Assertions.assertEquals(namespaceLocation + "/" + table, created.location());
}
+ @Test
Review Comment:
Maybe we can add a `testStagedCreateWithAppendFiles ` to verify the CTAS case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #4284:
URL: https://github.com/apache/amoro/pull/4284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
Posted by "johntomcat7408-cmyk (via GitHub)" <gi...@apache.org>.
johntomcat7408-cmyk commented on code in PR #4284:
URL: https://github.com/apache/amoro/pull/4284#discussion_r3680271266
##########
amoro-ams/src/main/java/org/apache/amoro/server/table/internal/InternalTableCreator.java:
##########
@@ -25,16 +25,19 @@
/** Interface to create an internal table. */
public interface InternalTableCreator extends Closeable {
- /**
- * Do all things about create an internal table, and prepare the {@link TableMetadata} for {@link
- * InternalTableManager#createTable(java.lang.String, TableMetadata)}
- */
+ /** Build Iceberg metadata without writing files or registering the table. */
+ org.apache.iceberg.TableMetadata stage();
Review Comment:
Both `org.apache.amoro.server.table.TableMetadata` and `org.apache.iceberg.TableMetadata` are used in this interface. Java cannot import both classes with the same simple name, so one of them must remain fully qualified. I kept the existing Amoro import because both `create` methods return that type; reversing the import would only move the FQCN.
##########
amoro-ams/src/main/java/org/apache/amoro/server/RestCatalogService.java:
##########
@@ -358,25 +366,101 @@ public void loadTable(Context ctx) {
/** POST PREFIX/v1/catalogs/{catalog}/namespaces/{namespace}/tables/{table} */
public void commitTable(Context ctx) {
- handleTable(
- ctx,
- handler -> {
- UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
- TableOperations ops = handler.newTableOperator();
- TableMetadata base = ops.current();
- if (base == null) {
- throw new CommitFailedException("table metadata lost.");
- }
+ UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
+ request.validate();
+ if (isCreate(request)) {
+ handleNamespace(
+ ctx,
+ (catalog, database) -> {
+ String tableName = ctx.pathParam("table");
+ Preconditions.checkNotNull(tableName, "table name is null");
+ return commitCreateTable(catalog, database, tableName, request);
+ });
+ } else {
+ handleTable(
+ ctx,
+ handler -> {
+ TableOperations ops = handler.newTableOperator();
+ TableMetadata base = ops.current();
+ if (base == null) {
+ throw new CommitFailedException("table metadata lost.");
+ }
- TableMetadata.Builder builder = TableMetadata.buildFrom(base);
- request.requirements().forEach(r -> r.validate(base));
- request.updates().forEach(u -> u.applyTo(builder));
- TableMetadata newMetadata = builder.build();
+ TableMetadata.Builder builder = TableMetadata.buildFrom(base);
+ request.requirements().forEach(r -> r.validate(base));
+ request.updates().forEach(u -> u.applyTo(builder));
+ TableMetadata newMetadata = builder.build();
- ops.commit(base, newMetadata);
- TableMetadata current = ops.current();
- return LoadTableResponse.builder().withTableMetadata(current).build();
- });
+ ops.commit(base, newMetadata);
+ TableMetadata current = ops.current();
+ return LoadTableResponse.builder().withTableMetadata(current).build();
+ });
+ }
+ }
+
+ private LoadTableResponse commitCreateTable(
+ InternalCatalog catalog, String database, String tableName, UpdateTableRequest request) {
+ request.requirements().forEach(requirement -> requirement.validate((TableMetadata) null));
+
+ Optional<Integer> formatVersion =
+ request.updates().stream()
+ .filter(MetadataUpdate.UpgradeFormatVersion.class::isInstance)
+ .map(MetadataUpdate.UpgradeFormatVersion.class::cast)
+ .map(MetadataUpdate.UpgradeFormatVersion::formatVersion)
+ .findFirst();
+ TableMetadata.Builder builder =
Review Comment:
Thanks for pointing this out. In Iceberg 1.7.2, `TableMetadata.buildFromEmpty()` starts at format v2, so `builder.upgradeFormatVersion(1)` would be a downgrade and fail for an explicit v1 request. `buildFromEmpty(int)` initializes the requested version; I made that overload selection explicit and added coverage for both default v2 and explicit v1.
##########
amoro-ams/src/main/java/org/apache/amoro/server/RestCatalogService.java:
##########
@@ -358,25 +366,101 @@ public void loadTable(Context ctx) {
/** POST PREFIX/v1/catalogs/{catalog}/namespaces/{namespace}/tables/{table} */
public void commitTable(Context ctx) {
- handleTable(
- ctx,
- handler -> {
- UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
- TableOperations ops = handler.newTableOperator();
- TableMetadata base = ops.current();
- if (base == null) {
- throw new CommitFailedException("table metadata lost.");
- }
+ UpdateTableRequest request = bodyAsClass(ctx, UpdateTableRequest.class);
+ request.validate();
+ if (isCreate(request)) {
+ handleNamespace(
+ ctx,
+ (catalog, database) -> {
+ String tableName = ctx.pathParam("table");
+ Preconditions.checkNotNull(tableName, "table name is null");
+ return commitCreateTable(catalog, database, tableName, request);
+ });
+ } else {
+ handleTable(
+ ctx,
+ handler -> {
+ TableOperations ops = handler.newTableOperator();
+ TableMetadata base = ops.current();
+ if (base == null) {
+ throw new CommitFailedException("table metadata lost.");
+ }
- TableMetadata.Builder builder = TableMetadata.buildFrom(base);
- request.requirements().forEach(r -> r.validate(base));
- request.updates().forEach(u -> u.applyTo(builder));
- TableMetadata newMetadata = builder.build();
+ TableMetadata.Builder builder = TableMetadata.buildFrom(base);
+ request.requirements().forEach(r -> r.validate(base));
+ request.updates().forEach(u -> u.applyTo(builder));
+ TableMetadata newMetadata = builder.build();
- ops.commit(base, newMetadata);
- TableMetadata current = ops.current();
- return LoadTableResponse.builder().withTableMetadata(current).build();
- });
+ ops.commit(base, newMetadata);
+ TableMetadata current = ops.current();
+ return LoadTableResponse.builder().withTableMetadata(current).build();
+ });
+ }
+ }
+
+ private LoadTableResponse commitCreateTable(
+ InternalCatalog catalog, String database, String tableName, UpdateTableRequest request) {
+ request.requirements().forEach(requirement -> requirement.validate((TableMetadata) null));
+
+ Optional<Integer> formatVersion =
+ request.updates().stream()
+ .filter(MetadataUpdate.UpgradeFormatVersion.class::isInstance)
+ .map(MetadataUpdate.UpgradeFormatVersion.class::cast)
+ .map(MetadataUpdate.UpgradeFormatVersion::formatVersion)
+ .findFirst();
+ TableMetadata.Builder builder =
+ formatVersion.map(TableMetadata::buildFromEmpty).orElseGet(TableMetadata::buildFromEmpty);
+ request.updates().forEach(update -> update.applyTo(builder));
+ TableMetadata icebergMetadata = builder.build();
+
+ CreateTableRequest createRequest =
Review Comment:
Added the comment. It now clarifies that the reconstructed `CreateTableRequest` only satisfies the current `InternalTableCreator` construction API, while `creator.create(icebergMetadata)` persists the fully reconstructed metadata.
##########
amoro-ams/src/test/java/org/apache/amoro/server/TestInternalIcebergCatalogService.java:
##########
@@ -284,6 +289,29 @@ public void testCreateTableInNamespaceLocation() {
Assertions.assertEquals(namespaceLocation + "/" + table, created.location());
}
+ @Test
Review Comment:
Added `testStagedCreateWithAppendFiles`. It writes real data files, appends them inside the staged transaction, verifies the table is still not registered before the final commit, and then verifies the committed snapshot and data-file paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-4240][AMS] Support staged table creation in REST catalog [amoro]
Posted by "johntomcat7408-cmyk (via GitHub)" <gi...@apache.org>.
johntomcat7408-cmyk commented on PR #4284:
URL: https://github.com/apache/amoro/pull/4284#issuecomment-5128439394
@zhoujinsong The latest Hadoop 3 / JDK 17 CI job failed in an unrelated optimizer test: `TestDefaultOptimizingService.testPollTaskThreeTimes` expected `null` but received a task after the test optimizer expired and its task was requeued.
I confirmed that the staged-create REST catalog tests changed by this PR passed in the same job, and this PR does not modify the optimizer service or that test. I filed #4303 to track the flaky heartbeat/timing condition separately.
Could you please rerun the failed CI workflow when convenient? I do not have permission to rerun it. The rerun result will help confirm whether #4303 is intermittent or consistently reproducible on CI.
Failed job: https://github.com/apache/amoro/actions/runs/30520803888/job/90800567957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org