You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "thomaschow (via GitHub)" <gi...@apache.org> on 2025/09/22 18:11:23 UTC
[PR] BigQuery: implement rename table [iceberg]
thomaschow opened a new pull request, #14159:
URL: https://github.com/apache/iceberg/pull/14159
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "thomaschow (via GitHub)" <gi...@apache.org>.
thomaschow commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3321569749
> have you tested this against the real api endpoint?
I have tested this in our canary environment against real biguery iceberg tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3321974982
Hi @thomaschow
BQMS support Managed Iceberg Tables and External Iceberg table. Your implementation only supports External tables. Managed Iceberg Table is not supported. Did you test with Managed Iceberg Tables?
Currently Managed Iceberg Tables does not export its metadata automatically. It requires `EXPORT TABLE METADATA` [statement](https://cloud.google.com/bigquery/docs/exporting-data#export_table_metadata). We are working on auto metadata export but it will be async. There is still potential data loss for Managed Iceberg Table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14159: BigQuery: implement rename table
URL: https://github.com/apache/iceberg/pull/14159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "thomaschow (via GitHub)" <gi...@apache.org>.
thomaschow commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3325622963
> Did you test with Managed Iceberg Tables?
I have not tested this flow. I was thinking supporting managed iceberg tables could be an incremental addition, since (correct me if I'm wrong) it would depend on a metadata export.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3499933389
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3677067163
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3648579022
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#discussion_r2370201143
##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +527,57 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
}
}
+ @Override
+ public void rename(TableReference fromTableReference, TableReference toTableReference) {
+ try {
+ Table sourceTable = load(fromTableReference);
+
+ DatasetReference toDatasetReference =
+ new DatasetReference()
+ .setProjectId(toTableReference.getProjectId())
+ .setDatasetId(toTableReference.getDatasetId());
+ load(toDatasetReference);
+
+ Table renamedTable = new Table();
+ renamedTable.setTableReference(toTableReference);
+ renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+ renamedTable.setSchema(sourceTable.getSchema());
+
+ try {
+ HttpResponse createResponse =
+ client
+ .tables()
+ .insert(
+ toTableReference.getProjectId(), toTableReference.getDatasetId(), renamedTable)
+ .executeUnparsed();
+
+ convertExceptionIfUnsuccessful(createResponse);
+
+ } catch (Exception createException) {
+ throw new RuntimeIOException(
+ "Failed to create new table %s during rename: %s",
+ toTableReference, createException.getMessage());
+ }
+
+ try {
+ delete(fromTableReference);
+ } catch (Exception deleteException) {
+ client
+ .tables()
+ .delete(
+ toTableReference.getProjectId(),
+ toTableReference.getDatasetId(),
+ toTableReference.getTableId())
+ .executeUnparsed();
Review Comment:
The cleanup delete operation in the catch block doesn't handle potential exceptions. If this delete fails, it could throw an exception that masks the original deleteException, making debugging difficult. Consider wrapping this cleanup operation in a try-catch block and logging any failures.
```suggestion
try {
client
.tables()
.delete(
toTableReference.getProjectId(),
toTableReference.getDatasetId(),
toTableReference.getTableId())
.executeUnparsed();
} catch (Exception cleanupException) {
System.err.println("Failed to cleanup newly created table " + toTableReference + " after failed rename: " + cleanupException.getMessage());
}
```
##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +527,57 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
}
}
+ @Override
+ public void rename(TableReference fromTableReference, TableReference toTableReference) {
+ try {
+ Table sourceTable = load(fromTableReference);
+
+ DatasetReference toDatasetReference =
+ new DatasetReference()
+ .setProjectId(toTableReference.getProjectId())
+ .setDatasetId(toTableReference.getDatasetId());
+ load(toDatasetReference);
+
+ Table renamedTable = new Table();
+ renamedTable.setTableReference(toTableReference);
+ renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+ renamedTable.setSchema(sourceTable.getSchema());
+
+ try {
+ HttpResponse createResponse =
+ client
+ .tables()
+ .insert(
+ toTableReference.getProjectId(), toTableReference.getDatasetId(), renamedTable)
+ .executeUnparsed();
+
+ convertExceptionIfUnsuccessful(createResponse);
+
+ } catch (Exception createException) {
+ throw new RuntimeIOException(
+ "Failed to create new table %s during rename: %s",
+ toTableReference, createException.getMessage());
+ }
+
+ try {
+ delete(fromTableReference);
+ } catch (Exception deleteException) {
+ client
+ .tables()
+ .delete(
+ toTableReference.getProjectId(),
+ toTableReference.getDatasetId(),
+ toTableReference.getTableId())
+ .executeUnparsed();
+ }
Review Comment:
The original deleteException is caught but not propagated or logged. Users won't know why the source table deletion failed. Consider logging the deleteException and then throwing it or including it in a more comprehensive error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#discussion_r2517063202
##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
}
}
+ @Override
+ public void rename(TableReference fromTableReference, TableReference toTableReference) {
+ try {
+ Table sourceTable = load(fromTableReference);
+
+ DatasetReference toDatasetReference =
+ new DatasetReference()
+ .setProjectId(toTableReference.getProjectId())
+ .setDatasetId(toTableReference.getDatasetId());
+ load(toDatasetReference);
+
+ Table renamedTable = new Table();
+ renamedTable.setTableReference(toTableReference);
+ renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
Review Comment:
I've recently tested BigLake metastore, and noticed it requires that the location must contain the table name. Does this BigQuery metastore have the same requirement?
##########
bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreClientImpl.java:
##########
@@ -527,6 +531,76 @@ public List<Tables> list(DatasetReference datasetReference, boolean listAllTable
}
}
+ @Override
+ public void rename(TableReference fromTableReference, TableReference toTableReference) {
+ try {
+ Table sourceTable = load(fromTableReference);
+
+ DatasetReference toDatasetReference =
+ new DatasetReference()
+ .setProjectId(toTableReference.getProjectId())
+ .setDatasetId(toTableReference.getDatasetId());
+ load(toDatasetReference);
+
+ Table renamedTable = new Table();
+ renamedTable.setTableReference(toTableReference);
+ renamedTable.setExternalCatalogTableOptions(sourceTable.getExternalCatalogTableOptions());
+
+ createNewTableDuringRename(toTableReference, renamedTable);
+ deleteOriginalTableDuringRename(fromTableReference, toTableReference);
Review Comment:
Does BigQuery preserve existing permissions even after drop & create?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] BigQuery: implement rename table [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #14159:
URL: https://github.com/apache/iceberg/pull/14159#issuecomment-3377839332
this PR generally lgtm, I will defer to @talatuyarer since im not familiar with the gcp ecosystem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org