You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "thomaschow (via GitHub)" <gi...@apache.org> on 2026/02/11 20:42:27 UTC

[PR] BigQuery: implement rename table [iceberg]

thomaschow opened a new pull request, #15300:
URL: https://github.com/apache/iceberg/pull/15300

   Re-creating https://github.com/apache/iceberg/pull/14159 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #15300:
URL: https://github.com/apache/iceberg/pull/15300#discussion_r2796013492


##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {

Review Comment:
   Does BigQuery metastore allow renaming a table across the namespace?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #15300:
URL: https://github.com/apache/iceberg/pull/15300#discussion_r2795802424


##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);

Review Comment:
   Does BigQuery preserve existing permissions even after drop & create?



##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());

Review Comment:
   I've recently tested BigLake metastore, and noticed it requires that the location must contain the table name. Does this BigQuery metastore have the same requirement?



##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);
+
+    } catch (Exception e) {
+      throw new RuntimeIOException(
+          "Failed to rename table from %s to %s: %s",
+          fromTableReference, toTableReference, e.getMessage());
+    }
+  }
+
+  private void createNewTableDuringRename(TableReference toTableReference, Table renamedTable) {
+    try {
+      HttpResponse createResponse =
+          client
+              .tables()
+              .insert(
+                  toTableReference.getProjectId(), toTableReference.getDatasetId(), renamedTable)
+              .executeUnparsed();
+
+      convertExceptionIfUnsuccessful(createResponse);
+
+    } catch (Exception createException) {
+      throw new RuntimeIOException(
+          "Failed to create new table %s during rename: %s",
+          toTableReference, createException.getMessage());
+    }
+  }
+
+  private void deleteOriginalTableDuringRename(
+      TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      delete(fromTableReference);
+    } catch (Exception deleteException) {
+      LOG.error("Failed to cleanup original table {}", fromTableReference, deleteException);
+      cleanupNewlyCreatedTableDuringRename(toTableReference);

Review Comment:
   This catch block seems unsafe. If an exception occurs on the client side (e.g., a timeout) after the remote operation has actually succeeded, both tables could end up being dropped, correct?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "thomaschow (via GitHub)" <gi...@apache.org>.
thomaschow closed pull request #15300: BigQuery: implement rename table
URL: https://github.com/apache/iceberg/pull/15300


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #15300:
URL: https://github.com/apache/iceberg/pull/15300#discussion_r2795802424


##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);

Review Comment:
   We'll lose the existing BigQuery permissions after we drop and recreate it, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "joyhaldar (via GitHub)" <gi...@apache.org>.
joyhaldar commented on code in PR #15300:
URL: https://github.com/apache/iceberg/pull/15300#discussion_r2802732616


##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);
+
+    } catch (Exception e) {
+      throw new RuntimeIOException(
+          "Failed to rename table from %s to %s: %s",
+          fromTableReference, toTableReference, e.getMessage());
+    }
+  }
+
+  private void createNewTableDuringRename(TableReference toTableReference, Table renamedTable) {

Review Comment:
   `BigQueryMetastoreClient` already has `create(Table)`. Could we reuse it instead of raw HTTP calls in `createNewTableDuringRename`?



##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {

Review Comment:
   Similar question as ebyhr, does BQMS support renaming tables across datasets? The current code validates both namespaces exist but doesn't check if they're the same.
   
   If cross-dataset renames aren't supported by the API, we could fail fast.
   
   If they are supported, there might be another concern, users who had access to the source dataset may not have access to the destination dataset, so they'd lose access to the table after rename.



##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);
+
+    } catch (Exception e) {
+      throw new RuntimeIOException(
+          "Failed to rename table from %s to %s: %s",
+          fromTableReference, toTableReference, e.getMessage());
+    }
+  }
+
+  private void createNewTableDuringRename(TableReference toTableReference, Table renamedTable) {
+    try {
+      HttpResponse createResponse =
+          client
+              .tables()
+              .insert(
+                  toTableReference.getProjectId(), toTableReference.getDatasetId(), renamedTable)
+              .executeUnparsed();
+
+      convertExceptionIfUnsuccessful(createResponse);
+
+    } catch (Exception createException) {
+      throw new RuntimeIOException(
+          "Failed to create new table %s during rename: %s",
+          toTableReference, createException.getMessage());
+    }
+  }
+
+  private void deleteOriginalTableDuringRename(
+      TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      delete(fromTableReference);
+    } catch (Exception deleteException) {
+      LOG.error("Failed to cleanup original table {}", fromTableReference, deleteException);
+      cleanupNewlyCreatedTableDuringRename(toTableReference);
+    }
+  }
+
+  private void cleanupNewlyCreatedTableDuringRename(TableReference toTableReference) {

Review Comment:
   `BigQueryMetastoreClient` already has `delete(TableReference)`. Could we reuse it instead of raw HTTP calls in `cleanupNewlyCreatedTableDuringRename`?



##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
     }
   }
 
+  @Override
+  public void rename(TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      Table sourceTable = load(fromTableReference);
+
+      DatasetReference toDatasetReference =
+          new DatasetReference()
+              .setProjectId(toTableReference.getProjectId())
+              .setDatasetId(toTableReference.getDatasetId());
+      load(toDatasetReference);
+
+      Table renamedTable = new Table();
+      renamedTable.setTableReference(toTableReference);
+      renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+      createNewTableDuringRename(toTableReference, renamedTable);
+      deleteOriginalTableDuringRename(fromTableReference, toTableReference);
+
+    } catch (Exception e) {
+      throw new RuntimeIOException(
+          "Failed to rename table from %s to %s: %s",
+          fromTableReference, toTableReference, e.getMessage());
+    }
+  }
+
+  private void createNewTableDuringRename(TableReference toTableReference, Table renamedTable) {
+    try {
+      HttpResponse createResponse =
+          client
+              .tables()
+              .insert(
+                  toTableReference.getProjectId(), toTableReference.getDatasetId(), renamedTable)
+              .executeUnparsed();
+
+      convertExceptionIfUnsuccessful(createResponse);
+
+    } catch (Exception createException) {
+      throw new RuntimeIOException(
+          "Failed to create new table %s during rename: %s",
+          toTableReference, createException.getMessage());
+    }
+  }
+
+  private void deleteOriginalTableDuringRename(
+      TableReference fromTableReference, TableReference toTableReference) {
+    try {
+      delete(fromTableReference);
+    } catch (Exception deleteException) {
+      LOG.error("Failed to cleanup original table {}", fromTableReference, deleteException);
+      cleanupNewlyCreatedTableDuringRename(toTableReference);

Review Comment:
   +1 on this concern. It might be worth looking at how other catalogs handle this scenario, could help inform the approach here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] BigQuery: implement rename table [iceberg]

Posted by "thomaschow (via GitHub)" <gi...@apache.org>.
thomaschow commented on PR #15300:
URL: https://github.com/apache/iceberg/pull/15300#issuecomment-4059805875

   Going to close this, it seems BigQuery metastore still has feature gaps relative to other open catalogs, we'll pursue an external catalog when integrating iceberg and bigquery. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org