You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Vino1016 (via GitHub)" <gi...@apache.org> on 2025/03/13 07:11:17 UTC
[I] Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration [iceberg]
Vino1016 opened a new issue, #12514:
URL: https://github.com/apache/iceberg/issues/12514
### Query engine
flink
### Question
Context:
We are currently using Apache Iceberg 0.14.0 (Hadoop Catalog + ORC format) and planning to upgrade to Iceberg 1.8.0 with AWS Glue Catalog + Parquet. Our goals are:
Migrate metadata management from Hadoop Catalog to AWS Glue Catalog.
Transition file formats from ORC to Parquet.
Ensure backward compatibility for existing tables during the upgrade process.
Key Questions:
Version Upgrade Path
Are there known challenges or best practices when upgrading from 0.14.0 to 1.8.0 directly? Should intermediate versions (e.g., 1.3.x) be prioritized for staged migration?
Catalog Migration
What is the recommended approach to migrate table metadata from Hadoop Catalog to Glue Catalog while maintaining continuous write operations on existing tables? Are there tools like MigrateTable or manual metadata adaptation required?
File Format Compatibility
Can a single Iceberg table support both ORC and Parquet files coexisting (e.g., historical data in ORC and new data in Parquet)? If yes, does this require explicit configuration (e.g., write.format.default)?
Backward Compatibility
How does Iceberg 1.8.0 handle reads/writes on tables originally created with 0.14.0? Are there specific upgrade steps (e.g., metadata JSON updates, partition spec evolution) to ensure seamless operation?
Community Experience
Has anyone successfully executed a similar upgrade path (Hadoop→Glue Catalog + ORC→Parquet)? Any critical pitfalls or tooling recommendations?
Environment:
Iceberg 0.14.0 → 1.8.0
Hadoop (HDFS) → AWS S3 + Glue Catalog
Mixed batch/streaming workloads (ongoing writes to legacy tables)
We appreciate any insights or references to documentation/community discussions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration [iceberg]
Posted by "Vino1016 (via GitHub)" <gi...@apache.org>.
Vino1016 commented on issue #12514:
URL: https://github.com/apache/iceberg/issues/12514#issuecomment-2795893690
> **Library Upgrade -**
>
> We don't have a specific upgrade guide here because the library is attempting to be as independent from the format as possible. This means that if your code compiles and runs with the new version of the library, we expect it to work identically with your table.
>
> So your main checks willl be around configuration changes and other performance sorts of things that have changed, but under the hood your table won't change.
>
> **File Format Switch -**
>
> You can have files in multiple formats in the same table, there shouldn't be anything you need to do.
>
> **Catalog Migration**
>
> This much more complicated. Doing this will require changing all of your clients to point to glue instead of using path based table access. This is most important for writers, when you have multiple catalogs working with the same metadata.json you essentially enter into a split-brain situation. I would probably make the glue table "read only" and use the "registerTable" api to set the current hadoop based metadata.json as the current table path using another process. Eventually when I was ready to switch over writers, I would stop the sync and point the writers at Glue.
> **图书馆升级 -**
>
> 我们这里没有具体的升级指南，因为该库试图尽可能地独立于格式。这意味着，如果您的代码使用新版本的库编译并运行，我们期望它与您的表格完全相同。
>
> 因此，您的主要检查将围绕配置更改和其他已更改的性能类型 ，但在底层您的表不会改变。
>
> **文件格式切换 -**
>
> 您可以在同一个表中拥有多种格式的文件，您不需要做任何事情。
>
> **目录迁移**
>
> 这要复杂得多。这样做需要将所有客户端改为指向 Glue，而不是使用基于路径的表访问。这对于写入器来说至关重要，因为当多个目录使用同一个 metadata.json 时，您实际上会陷入脑裂的情况。我可能会将 Glue 表设置为“只读”，并使用“registerTable”API 将当前基于 Hadoop 的 metadata.json 设置为使用另一个进程的当前表路径。最终，当我准备切换写入器时，我会停止同步并将写入器指向 Glue。
Thank you for your response. Through our testing, we found that while Glue Crawler could potentially address metadata management, we have concerns about the significant risks this might pose to our existing data. We've tentatively decided to upgrade to version 1.4.0 while retaining the original Hadoop Catalog instead of adopting Glue.
Regarding your point about Iceberg tables supporting both ORC and Parquet formats without additional adaptation, we observed that the file format is determined by the table properties set during table creation. However, we noticed that the Iceberg API does not provide a way to modify these properties after table creation. Could you clarify if there's a recommended approach to dynamically adjust the write format configuration (e.g., switching new data writes to Parquet while retaining existing ORC files), or would this require creating a new table with the desired properties?
Any insights would be greatly appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration [iceberg]
Posted by "Vino1016 (via GitHub)" <gi...@apache.org>.
Vino1016 closed issue #12514: Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration
URL: https://github.com/apache/iceberg/issues/12514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #12514:
URL: https://github.com/apache/iceberg/issues/12514#issuecomment-2722236595
We don't have a specific upgrade guide here because the library is attempting to be as independent from the format as possible. This means that if your code compiles and runs with the new version of the library, we expect it to work identically with your table.
So your main checks willl be around configuration changes and other performance sorts of things that have changed, but under the hood your table won't change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Seeking Guidance on Upgrading Iceberg from 0.14.0 to 1.8.0 with Catalog and File Format Migration [iceberg]
Posted by "Vino1016 (via GitHub)" <gi...@apache.org>.
Vino1016 commented on issue #12514:
URL: https://github.com/apache/iceberg/issues/12514#issuecomment-2795894546
> **Library Upgrade -**
>
> We don't have a specific upgrade guide here because the library is attempting to be as independent from the format as possible. This means that if your code compiles and runs with the new version of the library, we expect it to work identically with your table.
>
> So your main checks willl be around configuration changes and other performance sorts of things that have changed, but under the hood your table won't change.
>
> **File Format Switch -**
>
> You can have files in multiple formats in the same table, there shouldn't be anything you need to do.
>
> **Catalog Migration**
>
> This much more complicated. Doing this will require changing all of your clients to point to glue instead of using path based table access. This is most important for writers, when you have multiple catalogs working with the same metadata.json you essentially enter into a split-brain situation. I would probably make the glue table "read only" and use the "registerTable" api to set the current hadoop based metadata.json as the current table path using another process. Eventually when I was ready to switch over writers, I would stop the sync and point the writers at Glue.
Thank you for your response. Through our testing, we found that while Glue Crawler could potentially address metadata management, we have concerns about the significant risks this might pose to our existing data. We've tentatively decided to upgrade to version 1.4.0 while retaining the original Hadoop Catalog instead of adopting Glue.
Regarding your point about Iceberg tables supporting both ORC and Parquet formats without additional adaptation, we observed that the file format is determined by the table properties set during table creation. However, we noticed that the Iceberg API does not provide a way to modify these properties after table creation. Could you clarify if there's a recommended approach to dynamically adjust the write format configuration (e.g., switching new data writes to Parquet while retaining existing ORC files), or would this require creating a new table with the desired properties?
Any insights would be greatly appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org