You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/10/12 23:05:35 UTC
[GitHub] [iceberg] aokolnychyi opened a new issue #1591: Spark SQL Extensions: MIGRATE command
aokolnychyi opened a new issue #1591:
URL: https://github.com/apache/iceberg/issues/1591
One should be able to use the MIGRATE command to migrate existing tables to Iceberg. Similar to SNAPSHOT, it should use the existing table definition to create a new Iceberg table and generate metadata for existing files. Apart from that, it should either swap the table pointer in the original catalog or rename the original table to a backup table (depending on circumstances, let's discuss). Once the table has been migrated to Iceberg, all writes and reads have to be done through Iceberg. In other words, the original table should no longer be accessible to non-Iceberg readers.
```
MIGRATE TABLE t [AS t2]
USING iceberg
[TBLPROPERTIES ('key' 'value')]
```
In query engines like Spark where we have a notion of a custom catalog, we may not always be able to swap a pointer in the original catalog as the source and target catalogs may be different. For example, if you want to move a regular Spark table that stores a pointer in the HMS to the Iceberg Hadoop catalog. For such cases we may want to consider exposing AS target.
Important to note that MIGRATE should inherit the location of the original table. New files must be written in the same layout. For example, Iceberg must set the data location as the root table location whenever migrating an existing dataset as opposed to having a separate data folder. Users should be prohibited from modifying the data location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liukun4515 commented on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-718471616
> Is this issue addressed by PR #1525? @aokolnychyi @RussellSpitzer
I think #1525 just modify the action to support spark3( multi catalog).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-717271722
I will be making a subsequent PR that acknowledge adds the SQL but that PR is going to be the programmatic api which the command will use
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-754620422
This was done in PR #1906.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-707388311
@RussellSpitzer is looking into that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi closed issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
aokolnychyi closed issue #1591:
URL: https://github.com/apache/iceberg/issues/1591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] waterlx commented on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
waterlx commented on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-717105327
Is this issue addressed by PR #1525? @aokolnychyi @RussellSpitzer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer edited a comment on issue #1591: Spark SQL Extensions: MIGRATE command
Posted by GitBox <gi...@apache.org>.
RussellSpitzer edited a comment on issue #1591:
URL: https://github.com/apache/iceberg/issues/1591#issuecomment-717271722
I will be making a subsequent PR that adds the SQL but that PR #1525 is going to be the programmatic api which the command will use
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org