You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "talatuyarer (via GitHub)" <gi...@apache.org> on 2026/08/29 00:26:26 UTC
[PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
talatuyarer opened a new pull request, #17873:
URL: https://github.com/apache/iceberg/pull/17873
Adds the write path for Iceberg views to the `Flink v2.3`, following the read path in #17859 (part of #17858):
Following commands are implemented:
- `CREATE VIEW [IF NOT EXISTS]` with comment and column list
- `DROP VIEW [IF EXISTS]`
- `ALTER VIEW ... RENAME TO`
- `ALTER VIEW ... AS` now fails with a clear "Altering a view is not supported yet" instead of a misleading internal error; implementation is the next PR in the series.
`CREATE OR REPLACE VIEW` is intentionally absent. Flink (through 2.3) has no dispatch path for it on catalog views, so nothing ever reaches the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987109149
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -401,17 +468,48 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
toIdentifier(tablePath),
toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(
+ toIdentifier(tablePath),
+ toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ return;
+ } catch (NoSuchViewException viewException) {
+ e.addSuppressed(viewException);
+ } catch (AlreadyExistsException alreadyExistsException) {
+ throw new TableAlreadyExistException(
+ getName(),
+ new ObjectPath(tablePath.getDatabaseName(), newTableName),
Review Comment:
nit: Does this worth a variable?
`new ObjectPath(tablePath.getDatabaseName(), newTableName)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3995353765
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
Review Comment:
Added — the test asserts the exception type via the ValidationException cause and re-queries the table afterwards to show it was untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4007752961
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -507,6 +676,11 @@ private static void validateTablePartition(CatalogTable ct1, CatalogTable ct2) {
@Override
public void alterTable(ObjectPath tablePath, CatalogBaseTable newTable, boolean ignoreIfNotExists)
throws CatalogException, TableNotExistException {
+ if (newTable instanceof CatalogView) {
+ throw new UnsupportedOperationException(
+ "Altering a view is not supported yet for catalog: " + getName());
Review Comment:
Should this be simply:
```suggestion
"Altering a view is not supported.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987067943
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -473,6 +571,33 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
+ .withQuery(FLINK_DIALECT, view.getOriginalQuery())
+ .withProperties(properties)
+ .create();
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (NoSuchNamespaceException e) {
+ throw new CatalogException("Database does not exist: " + tablePath.getDatabaseName(), e);
Review Comment:
Should we throw `DatabaseNotExistException` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4008575255
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.Catalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Parameters(name = "catalogType={0}, baseNamespace={1}")
+ protected static List<Object[]> parameters() {
+ return Arrays.asList(
+ new Object[] {CatalogType.HIVE, Namespace.empty()},
+ new Object[] {CatalogType.HADOOP, Namespace.empty()},
+ new Object[] {CatalogType.REST, Namespace.empty()},
+ new Object[] {CatalogType.REST, Namespace.of("l0", "l1")});
+ }
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ Catalog flinkCatalog = getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReferenceFails() {
+ // the stored SQL resolves against the view's database, but the session validated it against
+ // db2 — accepting this would produce session-dependent results, so creation is rejected
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t",
+ catalogName, DATABASE))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("unqualified name");
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
+ sql("CREATE DATABASE %s.db2", catalogName);
+ try {
+ sql("CREATE TABLE %s.db2.other_t (id BIGINT)", catalogName);
+ sql("INSERT INTO %s.db2.other_t VALUES (9)", catalogName);
+ sql("CREATE VIEW cross_view AS SELECT id FROM db2.other_t");
+
+ assertSameElements(Lists.newArrayList(Row.of(9L)), sql("SELECT * FROM cross_view"));
+ // the result does not depend on the reader's session database
+ sql("USE db2");
+ assertSameElements(
+ Lists.newArrayList(Row.of(9L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.other_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewNotSupportedByCatalog() {
+ // a catalog without view support (HadoopCatalog) rejects CREATE VIEW with a clear error
+ String noViewCatalog = catalogName + "_noviews";
+ sql(
+ "CREATE CATALOG %s WITH ('type'='iceberg', 'catalog-type'='hadoop', 'warehouse'='file://%s/noviews')",
+ noViewCatalog, warehouseRoot());
+ try {
+ sql("CREATE DATABASE %s.no_view_db", noViewCatalog);
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.no_view_db.unsupported_view AS SELECT id, data FROM %s",
+ noViewCatalog, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Creating a view is not supported by catalog");
+ } finally {
+ dropDatabase(noViewCatalog + ".no_view_db", true);
+ dropCatalog(noViewCatalog, true);
+ }
+ }
+
+ @TestTemplate
+ public void testDropView() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+
+ sql("DROP VIEW %s", VIEW_NAME);
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ assertThat(viewCatalog().viewExists(TableIdentifier.of(icebergNamespace, VIEW_NAME))).isFalse();
+ }
+
+ @TestTemplate
+ public void testDropViewIfExists() {
+ sql("DROP VIEW IF EXISTS nonexistent_view");
+ assertThatThrownBy(() -> sql("DROP VIEW nonexistent_view"))
+ .hasMessageContaining("nonexistent_view");
Review Comment:
Added the exception type and full message check for the SQL path, and also a direct catalog API assertion for TableNotExistException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4008551109
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -378,40 +437,82 @@ private Table loadIcebergTable(ObjectPath tablePath) throws TableNotExistExcepti
@Override
public boolean tableExists(ObjectPath tablePath) throws CatalogException {
- return icebergCatalog.tableExists(toIdentifier(tablePath));
+ TableIdentifier identifier = toIdentifier(tablePath);
+ return icebergCatalog.tableExists(identifier)
+ || (canBeView(tablePath) && asViewCatalog.viewExists(identifier));
}
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; keep that
+ // behavior for tables and only consult the view catalog before returning
+ if (!dropped && canBeView(tablePath)) {
+ asViewCatalog.dropView(identifier);
Review Comment:
Good catch, fixed. Now it throws `TableNotExistException` when nothing was dropped and `ignoreIfNotExists` is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018433192
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987056263
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -378,18 +434,29 @@ private Table loadIcebergTable(ObjectPath tablePath) throws TableNotExistExcepti
@Override
public boolean tableExists(ObjectPath tablePath) throws CatalogException {
- return icebergCatalog.tableExists(toIdentifier(tablePath));
+ TableIdentifier identifier = toIdentifier(tablePath);
+ return icebergCatalog.tableExists(identifier)
+ || (canBeView(tablePath) && asViewCatalog.viewExists(identifier));
}
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
- throw new TableNotExistException(getName(), tablePath, e);
- }
+ dropped = false;
+ }
+
+ if (!dropped && canBeView(tablePath)) {
+ dropped = asViewCatalog.dropView(identifier);
+ }
+
+ if (!dropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
Review Comment:
Previously, we did not throw if the `dropped` was false for a table.
What is the expected behavior in Flink?
If this needs to be fixed, shall we do it in a different PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987072344
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -401,17 +468,48 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
toIdentifier(tablePath),
toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(
+ toIdentifier(tablePath),
+ toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ return;
+ } catch (NoSuchViewException viewException) {
+ e.addSuppressed(viewException);
+ } catch (AlreadyExistsException alreadyExistsException) {
+ throw new TableAlreadyExistException(
+ getName(),
+ new ObjectPath(tablePath.getDatabaseName(), newTableName),
+ alreadyExistsException);
+ }
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
} catch (AlreadyExistsException e) {
- throw new TableAlreadyExistException(getName(), tablePath, e);
+ throw new TableAlreadyExistException(
+ getName(), new ObjectPath(tablePath.getDatabaseName(), newTableName), e);
}
}
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
+ if (table instanceof CatalogView) {
+ if (asViewCatalog == null) {
+ throw new UnsupportedOperationException(
Review Comment:
Any chance to test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3899761421
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -331,9 +368,26 @@ public List<String> listTables(String databaseName)
}
@Override
- public CatalogTable getTable(ObjectPath tablePath)
+ public CatalogBaseTable getTable(ObjectPath tablePath)
throws TableNotExistException, CatalogException {
- Table table = loadIcebergTable(tablePath);
+ Table table;
+ try {
+ table = loadIcebergTable(tablePath);
+ } catch (TableNotExistException e) {
+ // metadata tables ("name$type") can never be views, and without a view catalog there is
+ // nothing else to look up
+ if (asViewCatalog == null || tablePath.getObjectName().contains("$")) {
Review Comment:
Extracted this into a `canBeView()` helper and applied it in `tableExists` and `dropTable` as well, so the view catalog is never consulted for metadata-table names anywhere. this also fixes tableExists potentially returning true for a name that getTable would refuse to load. Kept the check rather than dropping it: `$` is Flink's metadata-table syntax, so it should never resolve to a view, and skipping the lookup avoids an extra view-catalog call on failed metadata-table loads. The alter path is unchanged since CatalogManager resolves view identifiers before they reach the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#issuecomment-5666632975
The CI failure is due to a timeout while fetching or starting the S3-mock Docker container in Testcontainers. @pvary, could you please re-trigger the checks? Everything should pass on a fresh run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987127434
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining(VIEW_NAME);
Review Comment:
Maybe a bit more text to check on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3991419517
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -473,6 +571,33 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
Review Comment:
I have tested this and identified a session-related issue in here. In a session targeting `db2`, executing `CREATE VIEW db.v AS SELECT ... FROM t` (where `t` is in `db2`) is accepted. However, while the view works when read from the original `db2` session, it fails with "Object 't' not found" when accessed from any other session. This occurs because Flink's view expansion falls back to the reader's current database for any references missing from the view's database.
The root cause is that the Iceberg spec requires the `default-namespace` to match the session's namespace at view creation, which the Flink Catalog API does not natively expose. I have implemented explicit validation at creation time to handle this.
To ensure consistency, non-qualified references outside the view's catalog and database are now rejected with an error prompting for explicit qualification. Qualified cross-database references remain permitted, and I have added test cases to verify both scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987121886
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
Review Comment:
shall we check that the view is still there (not accidentally deleted)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3995349292
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -401,17 +468,48 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
toIdentifier(tablePath),
toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(
+ toIdentifier(tablePath),
+ toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ return;
+ } catch (NoSuchViewException viewException) {
+ e.addSuppressed(viewException);
+ } catch (AlreadyExistsException alreadyExistsException) {
+ throw new TableAlreadyExistException(
+ getName(),
+ new ObjectPath(tablePath.getDatabaseName(), newTableName),
+ alreadyExistsException);
+ }
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
} catch (AlreadyExistsException e) {
- throw new TableAlreadyExistException(getName(), tablePath, e);
+ throw new TableAlreadyExistException(
+ getName(), new ObjectPath(tablePath.getDatabaseName(), newTableName), e);
}
}
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
throws CatalogException, TableAlreadyExistException {
+ if (table instanceof CatalogView) {
+ if (asViewCatalog == null) {
+ throw new UnsupportedOperationException(
Review Comment:
Added testCreateViewNotSupportedByCatalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3894818129
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -331,9 +368,26 @@ public List<String> listTables(String databaseName)
}
@Override
- public CatalogTable getTable(ObjectPath tablePath)
+ public CatalogBaseTable getTable(ObjectPath tablePath)
throws TableNotExistException, CatalogException {
- Table table = loadIcebergTable(tablePath);
+ Table table;
+ try {
+ table = loadIcebergTable(tablePath);
+ } catch (TableNotExistException e) {
+ // metadata tables ("name$type") can never be views, and without a view catalog there is
+ // nothing else to look up
+ if (asViewCatalog == null || tablePath.getObjectName().contains("$")) {
Review Comment:
This check could be a method, and could be reused in `exists`, `drop` too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3915997750
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -401,6 +467,17 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
toIdentifier(tablePath),
toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(
+ toIdentifier(tablePath),
+ toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ return;
+ } catch (NoSuchViewException viewException) {
Review Comment:
Good catch! I added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4007756722
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.Catalog;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Parameters(name = "catalogType={0}, baseNamespace={1}")
+ protected static List<Object[]> parameters() {
+ return Arrays.asList(
+ new Object[] {CatalogType.HIVE, Namespace.empty()},
+ new Object[] {CatalogType.HADOOP, Namespace.empty()},
+ new Object[] {CatalogType.REST, Namespace.empty()},
+ new Object[] {CatalogType.REST, Namespace.of("l0", "l1")});
+ }
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ Catalog flinkCatalog = getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReferenceFails() {
+ // the stored SQL resolves against the view's database, but the session validated it against
+ // db2 — accepting this would produce session-dependent results, so creation is rejected
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t",
+ catalogName, DATABASE))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("unqualified name");
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
+ sql("CREATE DATABASE %s.db2", catalogName);
+ try {
+ sql("CREATE TABLE %s.db2.other_t (id BIGINT)", catalogName);
+ sql("INSERT INTO %s.db2.other_t VALUES (9)", catalogName);
+ sql("CREATE VIEW cross_view AS SELECT id FROM db2.other_t");
+
+ assertSameElements(Lists.newArrayList(Row.of(9L)), sql("SELECT * FROM cross_view"));
+ // the result does not depend on the reader's session database
+ sql("USE db2");
+ assertSameElements(
+ Lists.newArrayList(Row.of(9L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.other_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewNotSupportedByCatalog() {
+ // a catalog without view support (HadoopCatalog) rejects CREATE VIEW with a clear error
+ String noViewCatalog = catalogName + "_noviews";
+ sql(
+ "CREATE CATALOG %s WITH ('type'='iceberg', 'catalog-type'='hadoop', 'warehouse'='file://%s/noviews')",
+ noViewCatalog, warehouseRoot());
+ try {
+ sql("CREATE DATABASE %s.no_view_db", noViewCatalog);
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.no_view_db.unsupported_view AS SELECT id, data FROM %s",
+ noViewCatalog, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Creating a view is not supported by catalog");
+ } finally {
+ dropDatabase(noViewCatalog + ".no_view_db", true);
+ dropCatalog(noViewCatalog, true);
+ }
+ }
+
+ @TestTemplate
+ public void testDropView() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+
+ sql("DROP VIEW %s", VIEW_NAME);
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ assertThat(viewCatalog().viewExists(TableIdentifier.of(icebergNamespace, VIEW_NAME))).isFalse();
+ }
+
+ @TestTemplate
+ public void testDropViewIfExists() {
+ sql("DROP VIEW IF EXISTS nonexistent_view");
+ assertThatThrownBy(() -> sql("DROP VIEW nonexistent_view"))
+ .hasMessageContaining("nonexistent_view");
Review Comment:
Shall we match more? Type, or longer message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987124031
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
Review Comment:
shall we check that the table is still there (not accidentally deleted)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3995352032
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
Review Comment:
Added — the test now reloads the view after the IF NOT EXISTS statement and asserts it still has its original query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014104338
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
Review Comment:
could we update the comment and merge with the expanded query comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014168513
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
+ // the expanded query fully qualifies every table reference, so the view resolves the
+ // same way regardless of the reader's session; views built programmatically may not
+ // carry an expanded query, in which case the original query is stored as-is
+ .withQuery(FLINK_DIALECT, viewQuery(view))
+ .withProperties(properties)
+ .create();
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (NoSuchNamespaceException e) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName(), e);
+ }
+ }
+
+ private static String viewQuery(ResolvedCatalogView view) {
+ return StringUtils.isNullOrWhitespaceOnly(view.getExpandedQuery())
Review Comment:
Other places we use checks, like `comment != null && !comment.isEmpty()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4024561613
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -466,34 +467,86 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
+ Exception notFound = null;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
- throw new TableNotExistException(getName(), tablePath, e);
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry), others
+ // return false; either way consult the view catalog before deciding the object is missing
+ dropped = false;
+ notFound = e;
+ }
+
+ if (!dropped && canBeView(tablePath)) {
+ try {
+ dropped = asViewCatalog.dropView(identifier);
+ } catch (UnsupportedOperationException e) {
+ // the catalog rejects view operations at runtime, e.g. a JDBC catalog with a V0 schema
+ LOG.warn("Catalog {} rejects view operations; assuming no view support", getName(), e);
}
}
+
+ if (!dropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath, notFound);
+ }
}
@Override
public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ Preconditions.checkArgument(
+ !newTableName.contains("$"),
+ "Cannot rename %s to %s: '$' denotes a metadata table",
+ tablePath,
+ newTableName);
+
+ ObjectPath toPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
try {
- icebergCatalog.renameTable(
- toIdentifier(tablePath),
- toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ icebergCatalog.renameTable(toIdentifier(tablePath), toIdentifier(toPath));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
+ boolean renamed = false;
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(toIdentifier(tablePath), toIdentifier(toPath));
+ renamed = true;
+ } catch (NoSuchViewException viewException) {
+ e.addSuppressed(viewException);
+ } catch (UnsupportedOperationException viewException) {
+ // the catalog rejects view operations at runtime, e.g. a JDBC catalog with a V0 schema
+ e.addSuppressed(viewException);
+ } catch (AlreadyExistsException alreadyExistsException) {
+ throw new TableAlreadyExistException(getName(), toPath, alreadyExistsException);
+ }
+ }
+
+ if (!renamed && !ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
} catch (AlreadyExistsException e) {
- throw new TableAlreadyExistException(getName(), tablePath, e);
+ throw new TableAlreadyExistException(getName(), toPath, e);
}
}
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
- throws CatalogException, TableAlreadyExistException {
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ if (table instanceof CatalogView) {
+ if (asViewCatalog == null) {
+ throw new UnsupportedOperationException(
+ "Creating a view is not supported by catalog: " + getName());
+ }
+
+ Preconditions.checkArgument(
+ table instanceof ResolvedCatalogView,
+ "Expected a ResolvedCatalogView but got: %s",
+ table.getClass().getName());
+ createIcebergView(tablePath, (ResolvedCatalogView) table, ignoreIfExists);
Review Comment:
Do we need: canBeView?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018422359
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -333,6 +334,203 @@ public void testViewWithMatchingDefaultsIsReadable() {
assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
}
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // the expanded query is stored: every reference is fully qualified, so resolution does not
+ // depend on the reader's session
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(
+ String.format("FROM `%s`.`%s`.`%s`", catalogName, DATABASE, TABLE_NAME));
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReference() {
+ // the stored expanded query fully qualifies the reference at creation time, so the view
+ // resolves against the table the creator saw, regardless of the reader's session database
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ sql("INSERT INTO cross_t VALUES (7)");
+ sql("CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t", catalogName, DATABASE);
+
+ assertSameElements(
+ Lists.newArrayList(Row.of(7L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+
+ // reading from the view's own database, where an unqualified cross_t would not resolve
+ sql("USE %s", DATABASE);
+ assertSameElements(Lists.newArrayList(Row.of(7L)), sql("SELECT * FROM cross_view"));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
Review Comment:
No, reworded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3911903856
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -401,6 +467,17 @@ public void renameTable(ObjectPath tablePath, String newTableName, boolean ignor
toIdentifier(tablePath),
toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(
+ toIdentifier(tablePath),
+ toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ return;
+ } catch (NoSuchViewException viewException) {
Review Comment:
Shall we handle `AlreadyExistsException` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018425796
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -333,6 +334,203 @@ public void testViewWithMatchingDefaultsIsReadable() {
assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
}
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // the expanded query is stored: every reference is fully qualified, so resolution does not
+ // depend on the reader's session
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(
+ String.format("FROM `%s`.`%s`.`%s`", catalogName, DATABASE, TABLE_NAME));
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReference() {
+ // the stored expanded query fully qualifies the reference at creation time, so the view
+ // resolves against the table the creator saw, regardless of the reader's session database
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ sql("INSERT INTO cross_t VALUES (7)");
+ sql("CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t", catalogName, DATABASE);
+
+ assertSameElements(
+ Lists.newArrayList(Row.of(7L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+
+ // reading from the view's own database, where an unqualified cross_t would not resolve
+ sql("USE %s", DATABASE);
+ assertSameElements(Lists.newArrayList(Row.of(7L)), sql("SELECT * FROM cross_view"));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
+ sql("CREATE DATABASE %s.db2", catalogName);
+ try {
+ sql("CREATE TABLE %s.db2.other_t (id BIGINT)", catalogName);
+ sql("INSERT INTO %s.db2.other_t VALUES (9)", catalogName);
+ sql("CREATE VIEW cross_view AS SELECT id FROM db2.other_t");
+
+ assertSameElements(Lists.newArrayList(Row.of(9L)), sql("SELECT * FROM cross_view"));
+ // the result does not depend on the reader's session database
+ sql("USE db2");
+ assertSameElements(
+ Lists.newArrayList(Row.of(9L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.other_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewNotSupportedByCatalog() {
+ // a catalog without view support (HadoopCatalog) rejects CREATE VIEW with a clear error
+ String noViewCatalog = catalogName + "_noviews";
+ sql(
+ "CREATE CATALOG %s WITH ('type'='iceberg', 'catalog-type'='hadoop', 'warehouse'='file://%s/noviews')",
+ noViewCatalog, warehouseRoot());
+ try {
+ sql("CREATE DATABASE %s.no_view_db", noViewCatalog);
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.no_view_db.unsupported_view AS SELECT id, data FROM %s",
+ noViewCatalog, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Creating a view is not supported by catalog");
+ } finally {
+ dropDatabase(noViewCatalog + ".no_view_db", true);
+ dropCatalog(noViewCatalog, true);
+ }
+ }
+
+ @TestTemplate
+ public void testDropView() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+
+ sql("DROP VIEW %s", VIEW_NAME);
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ assertThat(viewCatalog().viewExists(TableIdentifier.of(icebergNamespace, VIEW_NAME))).isFalse();
+ }
+
+ @TestTemplate
+ public void testDropViewIfExists() {
+ sql("DROP VIEW IF EXISTS nonexistent_view");
+ assertThatThrownBy(() -> sql("DROP VIEW nonexistent_view"))
+ .isInstanceOf(ValidationException.class)
+ .hasMessageContaining("View with identifier")
+ .hasMessageContaining("nonexistent_view")
+ .hasMessageContaining("does not exist");
+
+ // dropping directly through the catalog API also reports the missing view
+ assertThatThrownBy(
Review Comment:
Split into `testDropNonexistentViewThroughCatalogApi`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018429158
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
Review Comment:
Restructured, no early returns anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4007728684
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -473,6 +574,74 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ /**
+ * The stored view SQL is later resolved against the view's own database, but Flink validated it
+ * against the session's current database. An unqualified reference to a table outside the view's
+ * database would therefore resolve differently (or not at all) depending on the reader's session,
+ * so such queries are rejected. Flink's expanded query fully qualifies every table reference,
+ * which makes the mismatch detectable without parsing SQL.
+ */
+ private void validateViewReferences(ObjectPath tablePath, ResolvedCatalogView view) {
+ String originalQuery = view.getOriginalQuery();
+ String expandedQuery = view.getExpandedQuery();
+ if (StringUtils.isNullOrWhitespaceOnly(originalQuery)
+ || StringUtils.isNullOrWhitespaceOnly(expandedQuery)) {
+ // views built programmatically may not carry Flink's normalized queries; the check only
+ // applies to parser-produced text
+ return;
+ }
+
+ Matcher references = QUALIFIED_REFERENCE.matcher(expandedQuery);
Review Comment:
This is "magic", but sometimes it does not work:
```
CREATE VIEW cat.db.mixed_v AS
SELECT x.id FROM db2.shared x JOIN shared y ON x.id = y.id;
```
We can try to patch this back and forth, but maybe using `getExpandedQuery` would be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4007741419
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -378,40 +437,82 @@ private Table loadIcebergTable(ObjectPath tablePath) throws TableNotExistExcepti
@Override
public boolean tableExists(ObjectPath tablePath) throws CatalogException {
- return icebergCatalog.tableExists(toIdentifier(tablePath));
+ TableIdentifier identifier = toIdentifier(tablePath);
+ return icebergCatalog.tableExists(identifier)
+ || (canBeView(tablePath) && asViewCatalog.viewExists(identifier));
}
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; keep that
+ // behavior for tables and only consult the view catalog before returning
+ if (!dropped && canBeView(tablePath)) {
+ asViewCatalog.dropView(identifier);
Review Comment:
Shall we check and throw if `!ignoreIfNotExists`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018454788
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
+ // the expanded query fully qualifies every table reference, so the view resolves the
+ // same way regardless of the reader's session; views built programmatically may not
+ // carry an expanded query, in which case the original query is stored as-is
+ .withQuery(FLINK_DIALECT, viewQuery(view))
+ .withProperties(properties)
+ .create();
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (NoSuchNamespaceException e) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName(), e);
+ }
+ }
+
+ private static String viewQuery(ResolvedCatalogView view) {
+ return StringUtils.isNullOrWhitespaceOnly(view.getExpandedQuery())
Review Comment:
That check went away with the fallback helper; `createIcebergView` calls `getExpandedQuery()` directly now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987130517
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining(TABLE_NAME);
Review Comment:
Maybe a bit more text to check on? Or the type of the exception?
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.flink;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assumptions.assumeThat;
+
+import java.util.List;
+import org.apache.flink.table.catalog.CatalogBaseTable;
+import org.apache.flink.table.catalog.CatalogView;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.TableAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.apache.flink.types.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.catalog.ViewCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.view.View;
+import org.apache.iceberg.view.ViewProperties;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+
+public class TestFlinkCatalogView extends CatalogTestBase {
+
+ private static final String TABLE_NAME = "test_table";
+ private static final String VIEW_NAME = "test_view";
+
+ private static final Schema VIEW_SCHEMA =
+ new Schema(
+ Types.NestedField.optional(1, "id", Types.LongType.get()),
+ Types.NestedField.optional(2, "data", Types.StringType.get()));
+
+ private static final Schema PROJECTED_VIEW_SCHEMA =
+ new Schema(Types.NestedField.optional(1, "id", Types.LongType.get()));
+
+ @Override
+ @BeforeEach
+ public void before() {
+ super.before();
+ assumeThat(isHadoopCatalog).as("HadoopCatalog does not implement ViewCatalog").isFalse();
+ sql("CREATE DATABASE %s", flinkDatabase);
+ sql("USE CATALOG %s", catalogName);
+ sql("USE %s", DATABASE);
+ sql("CREATE TABLE %s (id BIGINT, data STRING)", TABLE_NAME);
+ sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", TABLE_NAME);
+ }
+
+ @AfterEach
+ public void cleanNamespaces() {
+ if (validationCatalog instanceof ViewCatalog) {
+ ViewCatalog viewCatalog = (ViewCatalog) validationCatalog;
+ viewCatalog.listViews(icebergNamespace).forEach(viewCatalog::dropView);
+ }
+
+ sql("DROP TABLE IF EXISTS %s.%s", flinkDatabase, TABLE_NAME);
+ dropDatabase(flinkDatabase, true);
+ super.clean();
+ }
+
+ private ViewCatalog viewCatalog() {
+ assertThat(validationCatalog).isInstanceOf(ViewCatalog.class);
+ return (ViewCatalog) validationCatalog;
+ }
+
+ private View createView(String dialect, String query) {
+ return viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery(dialect, query)
+ .create();
+ }
+
+ private List<Row> expectedRows() {
+ return Lists.newArrayList(Row.of(1L, "a"), Row.of(2L, "b"), Row.of(3L, "c"));
+ }
+
+ @TestTemplate
+ public void testSelectFromView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromProjectedView() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM test_table")
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectFromViewWithQualifiedQuery() {
+ createView(
+ "flink", String.format("SELECT id, data FROM %s.%s.test_table", catalogName, DATABASE));
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSelectViewFromDifferentDatabase() {
+ // unqualified references in the stored SQL must resolve against the view's own database,
+ // not the session's current database (Flink expands views with the view's schema path)
+ createView("flink", "SELECT id, data FROM test_table");
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ assertSameElements(
+ expectedRows(), sql("SELECT * FROM %s.%s.%s", catalogName, DATABASE, VIEW_NAME));
+ } finally {
+ sql("USE %s", DATABASE);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testViewReferencingAnotherView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, "second_view"))
+ .withSchema(PROJECTED_VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id FROM " + VIEW_NAME)
+ .create();
+ assertSameElements(
+ Lists.newArrayList(Row.of(1L), Row.of(2L), Row.of(3L)), sql("SELECT * FROM second_view"));
+ }
+
+ @TestTemplate
+ public void testSqlForFallsBackToAnotherDialect() throws Exception {
+ // BaseView#sqlFor returns the first SQL representation when no "flink" one exists,
+ // e.g. a view created by Spark
+ createView("spark", "SELECT id, data FROM test_table");
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+
+ // ANSI SQL that both engines understand is directly usable
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testSqlForPrefersExactDialectMatch() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("spark", "SELECT id, data FROM spark_only_table")
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ CatalogView catalogView =
+ (CatalogView)
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getOriginalQuery()).isEqualTo("SELECT id, data FROM test_table");
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testViewCommentAndProperties() throws Exception {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .withProperty(ViewProperties.COMMENT, "view comment")
+ .withProperty("key1", "value1")
+ .create();
+
+ CatalogBaseTable catalogView =
+ getTableEnv().getCatalog(catalogName).get().getTable(new ObjectPath(DATABASE, VIEW_NAME));
+ assertThat(catalogView.getComment()).isEqualTo("view comment");
+ assertThat(catalogView.getOptions())
+ .containsEntry("key1", "value1")
+ .doesNotContainKey(ViewProperties.COMMENT);
+ }
+
+ @TestTemplate
+ public void testListViews() throws Exception {
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ assertThat(getTableEnv().getCatalog(catalogName).get().listViews(DATABASE))
+ .containsExactly(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testListTablesIncludesViews() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // Flink's Catalog#listTables contract covers both tables and views
+ assertThat(sql("SHOW TABLES")).containsExactlyInAnyOrder(Row.of(TABLE_NAME), Row.of(VIEW_NAME));
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testTableExistsForView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ org.apache.flink.table.catalog.Catalog flinkCatalog =
+ getTableEnv().getCatalog(catalogName).get();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, VIEW_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, TABLE_NAME))).isTrue();
+ assertThat(flinkCatalog.tableExists(new ObjectPath(DATABASE, "nonexistent"))).isFalse();
+ }
+
+ @TestTemplate
+ public void testViewNotExist() {
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent does not exist");
+ assertThatThrownBy(() -> sql("SELECT * FROM nonexistent"))
+ .isInstanceOf(org.apache.flink.table.api.ValidationException.class)
+ .hasMessageContaining("Object 'nonexistent' not found");
+ }
+
+ @TestTemplate
+ public void testMetadataTableNotRoutedToViewPath() {
+ createView("flink", "SELECT id, data FROM test_table");
+ // metadata table access must keep working and never hit the view branch
+ assertThat(sql("SELECT * FROM %s$snapshots", TABLE_NAME)).isNotEmpty();
+ assertThatThrownBy(
+ () ->
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .getTable(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isInstanceOf(TableNotExistException.class)
+ .hasMessageContaining("Table (or view) db.nonexistent$snapshots does not exist");
+ assertThat(
+ getTableEnv()
+ .getCatalog(catalogName)
+ .get()
+ .tableExists(new ObjectPath(DATABASE, "nonexistent$snapshots")))
+ .isFalse();
+ }
+
+ @TestTemplate
+ public void testDescribeView() {
+ createView("flink", "SELECT id, data FROM test_table");
+ assertThat(sql("DESCRIBE %s", VIEW_NAME))
+ .extracting(row -> row.getField(0))
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultNamespaceIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultNamespace(Namespace.of("some_other_db"))
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-namespace")
+ .hasMessageContaining("some_other_db");
+ }
+
+ @TestTemplate
+ public void testViewWithDifferentDefaultCatalogIsRejected() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog("some_other_catalog")
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertThatThrownBy(() -> sql("SELECT * FROM %s", VIEW_NAME))
+ .rootCause()
+ .hasMessageContaining("default-catalog")
+ .hasMessageContaining("some_other_catalog");
+ }
+
+ @TestTemplate
+ public void testViewWithMatchingDefaultsIsReadable() {
+ viewCatalog()
+ .buildView(TableIdentifier.of(icebergNamespace, VIEW_NAME))
+ .withSchema(VIEW_SCHEMA)
+ .withDefaultCatalog(catalogName)
+ .withDefaultNamespace(icebergNamespace)
+ .withQuery("flink", "SELECT id, data FROM test_table")
+ .create();
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // Flink hands the catalog its normalized (unparsed) SQL; references must stay unexpanded
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(String.format("FROM `%s`", TABLE_NAME))
+ .doesNotContain(catalogName);
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // without it, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
Review Comment:
Or the type of the exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018494683
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
+ boolean viewDropped = canBeView(tablePath) && asViewCatalog.dropView(identifier);
Review Comment:
Yes. dropTable now catches the `UnsupportedOperationException` from `dropView` and logs a warning, same as others. I also added `dropTable` and `renameTable` cases to `TestFlinkCatalogViewIncapableCatalog`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4027190988
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -466,34 +467,86 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
+ Exception notFound = null;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
- throw new TableNotExistException(getName(), tablePath, e);
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry), others
+ // return false; either way consult the view catalog before deciding the object is missing
+ dropped = false;
+ notFound = e;
+ }
+
+ if (!dropped && canBeView(tablePath)) {
+ try {
+ dropped = asViewCatalog.dropView(identifier);
+ } catch (UnsupportedOperationException e) {
+ // the catalog rejects view operations at runtime, e.g. a JDBC catalog with a V0 schema
+ LOG.warn("Catalog {} rejects view operations; assuming no view support", getName(), e);
}
}
+
+ if (!dropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath, notFound);
+ }
}
@Override
public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ Preconditions.checkArgument(
+ !newTableName.contains("$"),
+ "Cannot rename %s to %s: '$' denotes a metadata table",
+ tablePath,
+ newTableName);
+
+ ObjectPath toPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
try {
- icebergCatalog.renameTable(
- toIdentifier(tablePath),
- toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ icebergCatalog.renameTable(toIdentifier(tablePath), toIdentifier(toPath));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
+ boolean renamed = false;
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(toIdentifier(tablePath), toIdentifier(toPath));
+ renamed = true;
+ } catch (NoSuchViewException viewException) {
+ e.addSuppressed(viewException);
+ } catch (UnsupportedOperationException viewException) {
+ // the catalog rejects view operations at runtime, e.g. a JDBC catalog with a V0 schema
+ e.addSuppressed(viewException);
+ } catch (AlreadyExistsException alreadyExistsException) {
+ throw new TableAlreadyExistException(getName(), toPath, alreadyExistsException);
+ }
+ }
+
+ if (!renamed && !ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
} catch (AlreadyExistsException e) {
- throw new TableAlreadyExistException(getName(), tablePath, e);
+ throw new TableAlreadyExistException(getName(), toPath, e);
}
}
@Override
public void createTable(ObjectPath tablePath, CatalogBaseTable table, boolean ignoreIfExists)
- throws CatalogException, TableAlreadyExistException {
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ if (table instanceof CatalogView) {
+ if (asViewCatalog == null) {
+ throw new UnsupportedOperationException(
+ "Creating a view is not supported by catalog: " + getName());
+ }
+
+ Preconditions.checkArgument(
+ table instanceof ResolvedCatalogView,
+ "Expected a ResolvedCatalogView but got: %s",
+ table.getClass().getName());
+ createIcebergView(tablePath, (ResolvedCatalogView) table, ignoreIfExists);
Review Comment:
Added the same '$' check to the CREATE VIEW path with a test. Kept it separate from canBeView so "no view catalog" and "metadata table name" produce distinct errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#issuecomment-5700677759
Merged to main.
Thanks @talatuyarer for all the work you put into this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4008544998
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -473,6 +574,74 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ /**
+ * The stored view SQL is later resolved against the view's own database, but Flink validated it
+ * against the session's current database. An unqualified reference to a table outside the view's
+ * database would therefore resolve differently (or not at all) depending on the reader's session,
+ * so such queries are rejected. Flink's expanded query fully qualifies every table reference,
+ * which makes the mismatch detectable without parsing SQL.
+ */
+ private void validateViewReferences(ObjectPath tablePath, ResolvedCatalogView view) {
+ String originalQuery = view.getOriginalQuery();
+ String expandedQuery = view.getExpandedQuery();
+ if (StringUtils.isNullOrWhitespaceOnly(originalQuery)
+ || StringUtils.isNullOrWhitespaceOnly(expandedQuery)) {
+ // views built programmatically may not carry Flink's normalized queries; the check only
+ // applies to parser-produced text
+ return;
+ }
+
+ Matcher references = QUALIFIED_REFERENCE.matcher(expandedQuery);
Review Comment:
Yeah, you are right, your example breaks the check. 😄 I switched to storing getExpandedQuery() and removed the whole validation. Now every reference is qualified at creation time, so nothing depends on the reader's session anymore.
The trade-off is the catalog name ends up inside the stored SQL, so if someone registers the catalog under a different name the view fails with "catalog not found" but it is better loud failure instead of wrong results. I think this also answers your earlier default-namespace question, since the stored query does not rely on it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3991426510
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -378,18 +434,29 @@ private Table loadIcebergTable(ObjectPath tablePath) throws TableNotExistExcepti
@Override
public boolean tableExists(ObjectPath tablePath) throws CatalogException {
- return icebergCatalog.tableExists(toIdentifier(tablePath));
+ TableIdentifier identifier = toIdentifier(tablePath);
+ return icebergCatalog.tableExists(identifier)
+ || (canBeView(tablePath) && asViewCatalog.viewExists(identifier));
}
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
- throw new TableNotExistException(getName(), tablePath, e);
- }
+ dropped = false;
+ }
+
+ if (!dropped && canBeView(tablePath)) {
+ dropped = asViewCatalog.dropView(identifier);
+ }
+
+ if (!dropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#issuecomment-5638791399
I believe this is not blocker for our PR implementation but I have opened a new ticket to track the issue regarding view resolution for cross-database views: https://issues.apache.org/jira/browse/FLINK-40637 and send an email to Flink's dev list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014183069
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -333,6 +334,203 @@ public void testViewWithMatchingDefaultsIsReadable() {
assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
}
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // the expanded query is stored: every reference is fully qualified, so resolution does not
+ // depend on the reader's session
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(
+ String.format("FROM `%s`.`%s`.`%s`", catalogName, DATABASE, TABLE_NAME));
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReference() {
+ // the stored expanded query fully qualifies the reference at creation time, so the view
+ // resolves against the table the creator saw, regardless of the reader's session database
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ sql("INSERT INTO cross_t VALUES (7)");
+ sql("CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t", catalogName, DATABASE);
+
+ assertSameElements(
+ Lists.newArrayList(Row.of(7L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+
+ // reading from the view's own database, where an unqualified cross_t would not resolve
+ sql("USE %s", DATABASE);
+ assertSameElements(Lists.newArrayList(Row.of(7L)), sql("SELECT * FROM cross_view"));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
+ sql("CREATE DATABASE %s.db2", catalogName);
+ try {
+ sql("CREATE TABLE %s.db2.other_t (id BIGINT)", catalogName);
+ sql("INSERT INTO %s.db2.other_t VALUES (9)", catalogName);
+ sql("CREATE VIEW cross_view AS SELECT id FROM db2.other_t");
+
+ assertSameElements(Lists.newArrayList(Row.of(9L)), sql("SELECT * FROM cross_view"));
+ // the result does not depend on the reader's session database
+ sql("USE db2");
+ assertSameElements(
+ Lists.newArrayList(Row.of(9L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.other_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewNotSupportedByCatalog() {
+ // a catalog without view support (HadoopCatalog) rejects CREATE VIEW with a clear error
+ String noViewCatalog = catalogName + "_noviews";
+ sql(
+ "CREATE CATALOG %s WITH ('type'='iceberg', 'catalog-type'='hadoop', 'warehouse'='file://%s/noviews')",
+ noViewCatalog, warehouseRoot());
+ try {
+ sql("CREATE DATABASE %s.no_view_db", noViewCatalog);
+ assertThatThrownBy(
+ () ->
+ sql(
+ "CREATE VIEW %s.no_view_db.unsupported_view AS SELECT id, data FROM %s",
+ noViewCatalog, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Creating a view is not supported by catalog");
+ } finally {
+ dropDatabase(noViewCatalog + ".no_view_db", true);
+ dropCatalog(noViewCatalog, true);
+ }
+ }
+
+ @TestTemplate
+ public void testDropView() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ assertThat(sql("SHOW VIEWS")).containsExactly(Row.of(VIEW_NAME));
+
+ sql("DROP VIEW %s", VIEW_NAME);
+ assertThat(sql("SHOW VIEWS")).isEmpty();
+ assertThat(viewCatalog().viewExists(TableIdentifier.of(icebergNamespace, VIEW_NAME))).isFalse();
+ }
+
+ @TestTemplate
+ public void testDropViewIfExists() {
+ sql("DROP VIEW IF EXISTS nonexistent_view");
+ assertThatThrownBy(() -> sql("DROP VIEW nonexistent_view"))
+ .isInstanceOf(ValidationException.class)
+ .hasMessageContaining("View with identifier")
+ .hasMessageContaining("nonexistent_view")
+ .hasMessageContaining("does not exist");
+
+ // dropping directly through the catalog API also reports the missing view
+ assertThatThrownBy(
Review Comment:
Should this be a different test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014163036
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
+ // the expanded query fully qualifies every table reference, so the view resolves the
+ // same way regardless of the reader's session; views built programmatically may not
+ // carry an expanded query, in which case the original query is stored as-is
+ .withQuery(FLINK_DIALECT, viewQuery(view))
+ .withProperties(properties)
+ .create();
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (NoSuchNamespaceException e) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName(), e);
+ }
+ }
+
+ private static String viewQuery(ResolvedCatalogView view) {
+ return StringUtils.isNullOrWhitespaceOnly(view.getExpandedQuery())
+ ? view.getOriginalQuery()
+ : view.getExpandedQuery();
Review Comment:
Could this happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014295439
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
Review Comment:
I hate these `return` statements in the middle of the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #17873:
URL: https://github.com/apache/iceberg/pull/17873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#issuecomment-5700740203
Thank you @pvary to review and give feedbacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018465012
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
+ // the expanded query fully qualifies every table reference, so the view resolves the
+ // same way regardless of the reader's session; views built programmatically may not
+ // carry an expanded query, in which case the original query is stored as-is
+ .withQuery(FLINK_DIALECT, viewQuery(view))
+ .withProperties(properties)
+ .create();
+ } catch (AlreadyExistsException e) {
+ if (!ignoreIfExists) {
+ throw new TableAlreadyExistException(getName(), tablePath, e);
+ }
+ } catch (NoSuchNamespaceException e) {
+ throw new DatabaseNotExistException(getName(), tablePath.getDatabaseName(), e);
+ }
+ }
+
+ private static String viewQuery(ResolvedCatalogView view) {
+ return StringUtils.isNullOrWhitespaceOnly(view.getExpandedQuery())
+ ? view.getOriginalQuery()
+ : view.getExpandedQuery();
Review Comment:
Only if someone builds a `CatalogView` by hand with an empty expanded query. SQL always fills it, and a Table API view (`QueryOperationCatalogView`) has no SQL at all: it throws from `getOptions()` on the first line of `createIcebergView`. Dropped the fallback and use `getExpandedQuery()` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018475426
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
Review Comment:
Done. The expanded query already embeds the catalog name, so leaving `default-catalog` unset did not buy any portability anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987152848
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -473,6 +571,33 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
+ asViewCatalog
+ .buildView(toIdentifier(tablePath))
+ .withSchema(FlinkSchemaUtil.convert(view.getResolvedSchema()))
+ .withDefaultNamespace(appendLevel(baseNamespace, tablePath.getDatabaseName()))
Review Comment:
Are we sure that the `tablePath.getDatabaseName()` is correct here?
We need to sync the query and the namespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014112998
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -529,6 +573,42 @@ private boolean isReservedProperty(String prop) {
|| FlinkCreateTableOptions.SRC_CATALOG_PROPS_KEY.equalsIgnoreCase(prop);
}
+ private void createIcebergView(
+ ObjectPath tablePath, ResolvedCatalogView view, boolean ignoreIfExists)
+ throws CatalogException, DatabaseNotExistException, TableAlreadyExistException {
+ Map<String, String> properties = Maps.newHashMap(view.getOptions());
+ if (!StringUtils.isNullOrWhitespaceOnly(view.getComment())) {
+ properties.put(ViewProperties.COMMENT, view.getComment());
+ }
+
+ try {
+ // the default catalog is intentionally left unset so that readers resolve it to the
+ // name this catalog is registered under (see toCatalogView)
Review Comment:
Shall we `.withDefaultCatalog(getName())` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014174620
##########
flink/v2.3/flink/src/test/java/org/apache/iceberg/flink/TestFlinkCatalogView.java:
##########
@@ -333,6 +334,203 @@ public void testViewWithMatchingDefaultsIsReadable() {
assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
}
+ @TestTemplate
+ public void testCreateViewViaSql() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", VIEW_NAME));
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ // the expanded query is stored: every reference is fully qualified, so resolution does not
+ // depend on the reader's session
+ assertThat(view.sqlFor("flink").sql())
+ .containsIgnoringCase(
+ String.format("FROM `%s`.`%s`.`%s`", catalogName, DATABASE, TABLE_NAME));
+ assertThat(view.currentVersion().defaultNamespace()).isEqualTo(icebergNamespace);
+ assertThat(view.currentVersion().defaultCatalog()).isNull();
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("id", "data");
+ }
+
+ @TestTemplate
+ public void testCreateViewWithCommentAndColumnList() {
+ sql(
+ "CREATE VIEW %s (view_id, view_data) COMMENT 'a view comment' AS SELECT id, data FROM %s",
+ VIEW_NAME, TABLE_NAME);
+
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.properties()).containsEntry(ViewProperties.COMMENT, "a view comment");
+ assertThat(view.schema().columns())
+ .extracting(Types.NestedField::name)
+ .containsExactly("view_id", "view_data");
+ }
+
+ @TestTemplate
+ public void testCreateViewIfNotExists() {
+ sql("CREATE VIEW %s AS SELECT id, data FROM %s", VIEW_NAME, TABLE_NAME);
+ // IF NOT EXISTS is silent
+ sql("CREATE VIEW IF NOT EXISTS %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME);
+ // the view was not replaced: it still exists with its original query
+ View view = viewCatalog().loadView(TableIdentifier.of(icebergNamespace, VIEW_NAME));
+ assertThat(view.sqlFor("flink").sql()).containsIgnoringCase("data");
+
+ // without IF NOT EXISTS, creation fails
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", VIEW_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(VIEW_NAME);
+ }
+
+ @TestTemplate
+ public void testCreateViewOverExistingTableFails() {
+ assertThatThrownBy(() -> sql("CREATE VIEW %s AS SELECT id FROM %s", TABLE_NAME, TABLE_NAME))
+ .hasMessageContaining("Could not execute CreateTable")
+ .cause()
+ .isInstanceOf(TableAlreadyExistException.class)
+ .hasMessageContaining(TABLE_NAME);
+
+ // the table was not touched by the failed attempt
+ assertSameElements(expectedRows(), sql("SELECT * FROM %s", TABLE_NAME));
+ }
+
+ @TestTemplate
+ public void testCreateViewWithUnqualifiedCrossDatabaseReference() {
+ // the stored expanded query fully qualifies the reference at creation time, so the view
+ // resolves against the table the creator saw, regardless of the reader's session database
+ sql("CREATE DATABASE %s.db2", catalogName);
+ sql("USE db2");
+ try {
+ sql("CREATE TABLE cross_t (id BIGINT)");
+ sql("INSERT INTO cross_t VALUES (7)");
+ sql("CREATE VIEW %s.%s.cross_view AS SELECT id FROM cross_t", catalogName, DATABASE);
+
+ assertSameElements(
+ Lists.newArrayList(Row.of(7L)),
+ sql("SELECT * FROM %s.%s.cross_view", catalogName, DATABASE));
+
+ // reading from the view's own database, where an unqualified cross_t would not resolve
+ sql("USE %s", DATABASE);
+ assertSameElements(Lists.newArrayList(Row.of(7L)), sql("SELECT * FROM cross_view"));
+ } finally {
+ sql("USE %s", DATABASE);
+ sql("DROP TABLE IF EXISTS %s.db2.cross_t", catalogName);
+ dropDatabase(catalogName + ".db2", true);
+ }
+ }
+
+ @TestTemplate
+ public void testCreateViewWithQualifiedCrossDatabaseReference() {
+ // explicitly qualified references are deterministic and remain allowed
Review Comment:
Is this still true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4017803201
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
Review Comment:
Do we need to do the same check here as in the concurrent PR to prevent JDBC catalog to throw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4018393722
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
+ boolean viewDropped = canBeView(tablePath) && asViewCatalog.dropView(identifier);
+ if (!viewDropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
}
}
@Override
public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ ObjectPath toPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
try {
- icebergCatalog.renameTable(
- toIdentifier(tablePath),
- toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ icebergCatalog.renameTable(toIdentifier(tablePath), toIdentifier(toPath));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(toIdentifier(tablePath), toIdentifier(toPath));
Review Comment:
Good point. renameTable now rejects a target containing $ up front, for tables and views alike. Added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014067645
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
+ boolean viewDropped = canBeView(tablePath) && asViewCatalog.dropView(identifier);
+ if (!viewDropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
+ }
}
}
@Override
public void renameTable(ObjectPath tablePath, String newTableName, boolean ignoreIfNotExists)
throws TableNotExistException, TableAlreadyExistException, CatalogException {
+ ObjectPath toPath = new ObjectPath(tablePath.getDatabaseName(), newTableName);
try {
- icebergCatalog.renameTable(
- toIdentifier(tablePath),
- toIdentifier(new ObjectPath(tablePath.getDatabaseName(), newTableName)));
+ icebergCatalog.renameTable(toIdentifier(tablePath), toIdentifier(toPath));
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ if (canBeView(tablePath)) {
+ try {
+ asViewCatalog.renameView(toIdentifier(tablePath), toIdentifier(toPath));
Review Comment:
shall we reject target metadata tables? (renaming to `shadow$snapshots`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r4014071874
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -440,34 +441,77 @@ public boolean tableExists(ObjectPath tablePath) throws CatalogException {
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
+ // some catalogs signal "not a table" by throwing (e.g. Hive for a view entry)
+ if (canBeView(tablePath) && asViewCatalog.dropView(identifier)) {
+ return;
+ }
+
if (!ignoreIfNotExists) {
throw new TableNotExistException(getName(), tablePath, e);
}
+
+ return;
+ }
+
+ // others return false for a missing table; consult the view catalog before
+ // deciding the object does not exist
+ if (!dropped) {
+ boolean viewDropped = canBeView(tablePath) && asViewCatalog.dropView(identifier);
Review Comment:
Do we need to do the same check here as in the concurrent PR to prevent JDBC catalog to throw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3987060445
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -378,18 +434,29 @@ private Table loadIcebergTable(ObjectPath tablePath) throws TableNotExistExcepti
@Override
public boolean tableExists(ObjectPath tablePath) throws CatalogException {
- return icebergCatalog.tableExists(toIdentifier(tablePath));
+ TableIdentifier identifier = toIdentifier(tablePath);
+ return icebergCatalog.tableExists(identifier)
+ || (canBeView(tablePath) && asViewCatalog.viewExists(identifier));
}
@Override
public void dropTable(ObjectPath tablePath, boolean ignoreIfNotExists)
throws TableNotExistException, CatalogException {
+ TableIdentifier identifier = toIdentifier(tablePath);
+
+ boolean dropped;
try {
- icebergCatalog.dropTable(toIdentifier(tablePath));
+ dropped = icebergCatalog.dropTable(identifier);
} catch (org.apache.iceberg.exceptions.NoSuchTableException e) {
- if (!ignoreIfNotExists) {
- throw new TableNotExistException(getName(), tablePath, e);
- }
+ dropped = false;
+ }
+
+ if (!dropped && canBeView(tablePath)) {
+ dropped = asViewCatalog.dropView(identifier);
+ }
+
+ if (!dropped && !ignoreIfNotExists) {
+ throw new TableNotExistException(getName(), tablePath);
Review Comment:
Also chain the original exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Flink: Support CREATE VIEW, DROP VIEW and ALTER VIEW RENAME in FlinkCatalog [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #17873:
URL: https://github.com/apache/iceberg/pull/17873#discussion_r3894833038
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java:
##########
@@ -331,9 +368,26 @@ public List<String> listTables(String databaseName)
}
@Override
- public CatalogTable getTable(ObjectPath tablePath)
+ public CatalogBaseTable getTable(ObjectPath tablePath)
throws TableNotExistException, CatalogException {
- Table table = loadIcebergTable(tablePath);
+ Table table;
+ try {
+ table = loadIcebergTable(tablePath);
+ } catch (TableNotExistException e) {
+ // metadata tables ("name$type") can never be views, and without a view catalog there is
+ // nothing else to look up
+ if (asViewCatalog == null || tablePath.getObjectName().contains("$")) {
Review Comment:
Or just ignore this check here too, and try anyway... My main point it is not consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org