You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2025/04/10 12:19:07 UTC
[I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
slfan1989 opened a new issue, #12762:
URL: https://github.com/apache/iceberg/issues/12762
### Feature Request / Improvement
> Background
We are currently adopting Iceberg technology to drive internal technical iterations. One of the major challenges we face is migrating over 5000 Hive tables to Iceberg, some of which are extremely large. After researching the two migration methods provided by Iceberg (SnapShot and Migrate), we found that both have some shortcomings.
The migration methods supported by Iceberg are as follows:
- SnapShot: This method creates a new Iceberg table and maps the data files from the old table. Users are required to use the migrated Iceberg table.
- Migrate: This method generates an Iceberg table with the same name as the source table, but during the migration, write operations on the source table must be stopped, and data cannot be read during the migration.
The shortcomings of these two methods are:
- SnapShot: Users need to change the table name, which can have a significant impact on tasks with complex dependencies, especially when the source table has many downstream tasks, resulting in a heavy burden on users for reengineering.
- Migrate: During the migration, the source table cannot be read, and both read and write operations must be interrupted. The migration time depends on the size of the table, and for extremely large tables with thousands or tens of thousands of partitions, the migration time can be quite long. During this period, users cannot access the source table, which may disrupt other business operations.
To address these issues, we propose a new migration method that combines the advantages of both SnapShot and Migrate. In this solution, the source table remains readable during the migration process, and the table name of the source table remains unchanged after the upgrade. This makes the entire migration process completely transparent to users, with no impact on business operations.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2960996181
I’m really looking forward to hearing your thoughts. From my personal perspective, I truly hope to drive this approach into the Iceberg community, as it would provide a much more efficient way to migrate from Hive to Iceberg.
This approach is not only lightweight but also very easy to roll back, making it almost a seamless migration. With this method, we can help more users transition directly from Hive tables to Iceberg tables, enhancing the convenience and efficiency of the entire migration process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-3900263473
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2812274968
@deniskuzZ Thank you for your message! I am currently testing the approach recommended by @pvary locally, covering various scenarios such as tables with many partitions, multi-level partition tables, and non-partitioned tables. I plan to compile a comprehensive test report after some time, which I hope will serve as a reference and help for other developers with Hive2Iceberg needs. However, from my personal perspective, I believe we still need a Spark stored procedure to support the migration. Since Spark's transformations are distributed, theoretically, it could be faster for handling large tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12762: Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade
URL: https://github.com/apache/iceberg/issues/12762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2960989911
During the migration process, we encountered several challenges and addressed them with the following solutions:
- Rollback of Migrated Hive Tables
After migration, users may use the production tables in different ways. Sometimes, due to long periods without system updates, the system cannot immediately support accessing data through Iceberg tables. For example, partition access through show partitions db.table may no longer be supported and users need to switch to select * from db.table.partitions. As a result, users wanted the ability to roll back the migrated table to a Hive table.
To address this, we introduced a new stored procedure, revert, which directly removes the Iceberg metadata on the HMS side. After the rollback, the user only needs to re-execute the production tasks for a specific period. For example, if Table A was migrated on June 2, 2025, and a rollback is required on June 5, 2025, after using the stored procedure for the rollback, the user would simply need to re-run the scheduling tasks for June 3 to June 5, 2025, and Table A will revert to a Hive table.
- Aligning Partition Layout with Hive Table
Typically, Hive table partitions are stored in the format /dbName/tableName/dt=20250610/, while Iceberg partitions follow the format /dbName/tableName/dt=20250610/data/. To ensure that the migrated Iceberg tables retain the original partition layout, we added a configuration option that allows Iceberg tables to continue using the original Hive partition layout after migration.
- Limited Support for Using Iceberg Tables with Spark 2
The community's support for Iceberg in Spark 3 is very robust, and we have migrated 80% of our tasks to Spark 3. However, there are still some tasks that depend on Spark 2 due to business constraints, and migrating these tasks is difficult. Despite this, users still wish to use Iceberg tables with Spark 2. In such cases, we agreed with users to only support reading Iceberg tables in Spark 2, while also considering partition pushdown scenarios. We have implemented basic partition pushdown capabilities, and while this code may have limited impact on the Iceberg community, I plan to release it in a personal repository and provide a link for users with similar needs to refer to.
- Supporting More Partition Pushdown Scenarios
Some of our tables have multi-level partitions, such as year, month, and day. Writing complex partition filter conditions for these types of tables can be difficult. For example, users might need to filter data between 2023-01-01 and 2025-12-31, which results in a cumbersome partition expression. To solve this, we introduced a syntax similar to concat(year, month, day) >= '2023-01-01' and concat(year, month, day) <= '2025-12-31'. We defined a multi-level partition pushdown filtering expression in Iceberg and converted Spark's filters into partition pushdown expressions, which simplified the user's queries.
These optimizations have effectively addressed the challenges encountered during the migration process, ensuring a smooth transition to Iceberg for users while enhancing the flexibility and efficiency of the system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "navinvishy (via GitHub)" <gi...@apache.org>.
navinvishy commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-3194542659
@slfan1989 we are also looking into this currently, and are facing issues similar to yours. Thanks for working on this. I had a related question: for tables with a very large number of partitions, we frequently move older partitions into infrequent access storage classes on S3 such as deep archive. From what I can tell from the snapshot/migrate procedures, though they do not scan the entire files, they still require [reading footers](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java#L227) for metadata. This would be prohibitively expensive for files in deep archive. Is it possible to disable this for iceberg and collect no metadata? Could that be made an option in iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2812214850
> Couldn't https://issues.apache.org/jira/browse/HIVE-25008 work for you?
>
> Since Hive 4.0.0 the tables could be migrated directly to an Iceberg table with a simple alter command:
>
> ```
> ALTER TABLE ... SET TBLPROPERTIES('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler')
> ```
There is a much simpler SQL `ALTER TABLE ... CONVERT TO iceberg`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-3978701426
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2792670266
@pvary Thank you for your response! This is a very good solution. We have also implemented a solution based on Spark Procedures internally. However, our changes are relatively smaller, as they are derived from modifications to the SnapShot Procedure and RewriteTablePath Procedure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2795599098
@pvary Although Hive provides a good solution, after careful consideration, I still hope to seize an opportunity to implement a similar functionality in Iceberg’s Spark Procedure, offering more choices to users.
From a user’s perspective, our online environment uses Hive version 2.3.9, and upgrading to a new version is somewhat challenging due to the need for extensive compatibility evaluations. Additionally, there are significant code differences between Hive 2.3.9 and Hive 4.0.0. Therefore, I can only try to backport the relevant PRs to our internal branch following the approach of HIVE-25008. Since Hive’s HMS is a stateful service, the upgrade process will take some time.
Considering that 90% of our SQL tasks run on Spark, and the changes to Iceberg will eventually be compiled into a Spark runtime package, we can quickly update the Spark client to implement this functionality. After a comprehensive evaluation, using Spark’s stored procedure seems to be the more straightforward option.
To facilitate discussion, I will submit a draft PR that outlines the implementation approach. The changes are relatively minor and primarily reuse existing Iceberg capabilities.
cc: @nastra @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2960987469
@szehon-ho @manuzhang @pvary @deniskuzZ @dramaticlly
> Conclusion
Compared to Hive, Spark offers significant advantages, especially in high-concurrency and large-scale migration scenarios. Its lightweight operation, low system load, high transparency, and simple rollback mechanism make it the preferred choice for our table migration. To date, we have successfully migrated over 500 tables using Spark, and we expect this number to exceed 1,000 by the end of the month.
> Migration Requirements and Background:
During the migration process, we perform batch migrations based on the specific tasks of users, adhering to the following requirements:
Migration Transparency: The migration process must be fully transparent to the user, and the original tables must remain available without downtime.
Time Sensitivity: For daily-scheduled tasks, the migration must be completed within 1 day; for hourly-scheduled tasks, the migration must not exceed 1 hour.
> Challenges with Hive Migration:
- Heavy Load in High Concurrency: When migrating 30-40 tables concurrently, especially when the tables have thousands of partitions, or even over 10,000 partitions, the partition conversion efficiency using HMS is very low. This increases the load on HMS and may affect the response times of other queries.
- DDL Operation Risks: For tables with many partitions, users are typically hesitant to execute DDL operations directly due to concerns about inconsistent states or potential disruptions during the migration. This necessitates manual intervention by administrators, adding complexity and risk to the migration process.
> Advantages of Spark Migration:
- Lightweight Operation: Spark's migration process is very lightweight. During snapshot creation, it does not impact the original table, and metadata migration does not affect the table's usage. The only operation required is updating the HMS metadata and writing the metajson file. To further reduce the load on HMS, we configure multiple HMS instances and use a polling mechanism, significantly lowering the load even for large tables.
- No Concerns About Table Modifications: Since Spark’s migration process has little to no impact on the original table, users can continue to operate on the table during the migration. This provides greater flexibility and control to users, ensuring full transparency throughout the migration process.
- Simple Rollback Mechanism: In case of any issues during migration, the rollback process is straightforward. Since no changes are made to the user data, the only necessary action is to delete the metadata file and restore the Hive metadata configuration, eliminating the complexity and potential risks of data recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2792599740
Couldn't https://issues.apache.org/jira/browse/HIVE-25008 work for you?
Since Hive 4.0.0 the tables could be migrated directly to an Iceberg table with a simple alter command:
```
ALTER TABLE ... SET TBLPROPERTIES('storage_handler'='org.apache.iceberg.mr.hive.HiveIcebergStorageHandler')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Introducing a New Hive to Iceberg Table Migration Method: In-Place Upgrade [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #12762:
URL: https://github.com/apache/iceberg/issues/12762#issuecomment-2792582933
@nastra @szehon-ho Could you please take a look at this issue? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org