You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Gowthami03B (via GitHub)" <gi...@apache.org> on 2024/05/02 18:23:28 UTC
[I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]
Gowthami03B opened a new issue, #10262:
URL: https://github.com/apache/iceberg/issues/10262
### Feature Request / Improvement
Hello
The current snapshot procedure (https://iceberg.apache.org/docs/nightly/spark-procedures/?h=spark_catalog#snapshot) seems to be helpful in only migrating from external Hive to iceberg tables.
But we have a unique use case where we want to **migrate** _some of our tables from one namespace to another_ and later run 'alter schema operations' (which is metadata only) that would have worked perfectly for us with the "snapshot" procedure since it utilizes the underlying data files while having the new table's metadata in a new location.
The rest of tables in the old namespace would have to be backfilled as we have major changes, but we would avoid a bunch of effort and storage space(talking TB's here) if we could use **snapshot** procedure.
```
spark_jdbc_config = {
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
"spark.sql.catalog.my_catalog": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.my_catalog.catalog-impl": "org.apache.iceberg.jdbc.JdbcCatalog",
"spark.sql.catalog.my_catalog.uri": "jdbc:comdb2://",
"spark.sql.catalog.my_catalog.warehouse": "s3a://abc",
}
spark.sql(
f"""
CALL my_catalog.system.snapshot(
source_table => 'ns1.src_dataset' **# SparkConnectGrpcException: (org.apache.iceberg.exceptions.NoSuchTableException) Cannot not find source table 'datasets.equitynamr'**
table => 'ns2.src_dataset',
location => 's3a://abc'
)
"""
)
```
my_catalog here is the JDBC catalog that holds both the namespaces ns1, ns2 and all of our tables.
When I try to provide source_table as fully qualified name (my_catalog.ns1.src_dataset), I get this -IllegalArgumentException: Cannot snapshot a table that isn't in the session catalog (i.e. spark_catalog). Found source catalog: test.
I also tried explicitly creating a table with a catalog entry for 'spark_catalog', and that resulted in - IllegalArgumentException: Cannot use non-v1 table 'ns1.src_datasets' as a source
Is there any workaround to achieve my use case? Does this seem like a valid request that can be accommodated?
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10262:
URL: https://github.com/apache/iceberg/issues/10262#issuecomment-2540264645
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10262:
URL: https://github.com/apache/iceberg/issues/10262#issuecomment-2564111395
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]
Posted by "gauthamnair (via GitHub)" <gi...@apache.org>.
gauthamnair commented on issue #10262:
URL: https://github.com/apache/iceberg/issues/10262#issuecomment-2169023746
We are experiencing the same problem: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1718395195306979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] SnapshotTableProcedure to migrate iceberg tables from one namespace to another [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10262: SnapshotTableProcedure to migrate iceberg tables from one namespace to another
URL: https://github.com/apache/iceberg/issues/10262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org