You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dramaticlly (via GitHub)" <gi...@apache.org> on 2024/12/11 00:48:55 UTC
Re: [I] Rest Catalog: spark catalog api fails to work with rest based catalog [iceberg]
dramaticlly commented on issue #11741:
URL: https://github.com/apache/iceberg/issues/11741#issuecomment-2533365254
I think use this we can reproduce the problem in a unit test from `org.apache.iceberg.spark.sql.TestCreateTable` as it support all of hive/spark/hadoop/rest catalog, I think rest and hadoop catalog will fail while hive and spark will pass
```java
@TestTemplate
public void testCreateTable() {
assumeThat(catalogName).isEqualTo(SparkCatalogConfig.REST.catalogName());
assertThat(validationCatalog.tableExists(tableIdent))
.as("Table should not already exist")
.isFalse();
sql("CREATE TABLE %s (id BIGINT NOT NULL, data STRING) USING iceberg", tableName);
Table table = validationCatalog.loadTable(tableIdent);
assertThat(table).as("Should load the new table").isNotNull();
StructType expectedSchema =
StructType.of(
NestedField.required(1, "id", Types.LongType.get()),
NestedField.optional(2, "data", Types.StringType.get()));
assertThat(table.schema().asStruct())
.as("Should have the expected schema")
.isEqualTo(expectedSchema);
assertThat(table.spec().fields()).as("Should not be partitioned").hasSize(0);
assertThat(table.properties().get(TableProperties.DEFAULT_FILE_FORMAT))
.as("Should not have the default format set")
.isNull();
spark.sessionState().catalogManager().setCurrentCatalog(catalogName);
assertThat(spark.catalog().tableExists(tableIdent.toString())).isTrue(); // success
assertThat(spark.catalog().tableExists(tableIdent.namespace().toString(), tableIdent.name())).isTrue(); //failure
}
```
I am wondering if anyone run into such when using REST base catalog?
CC @RussellSpitzer @flyrain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org