You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "JonasJ-ap (via GitHub)" <gi...@apache.org> on 2023/01/22 05:41:23 UTC
[GitHub] [iceberg] JonasJ-ap opened a new pull request, #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
JonasJ-ap opened a new pull request, #6642:
URL: https://github.com/apache/iceberg/pull/6642
This PR is under construction, but I want to put it here for some initial feedback and discussion about the conversion from Apache Hudi to Apache Iceberg
## Overview
This PR aims to add a module called `iceberg-hudi` which contains public API and a base implementation to snapshot a hudi table to iceberg table. In expectation, the base implementation should rely on `hudi-common` module to extract metadata, timeline, locations of datafiles, and other information necessary for the conversion.
`copy-on-write`(COW) and `merge-on-read` (MOR) are two types of hudi table. As the initial implementation, this PR will focus on the conversion logic for COW tables.
The overall structure of the module is expected to be similar to #6449 . However, things may change as hudi is different from the delta lake. Also, due to the complexity of the conversion, I may make a proposal later for further discussion in the community.
## High-level Ideas
The base implementation of the snapshot action involves schema conversion and timeline replay. The idea here is to map every completed `COMMIT` action on the timeline to an iceberg snapshot. The conception of `COW` can be mapped to the `overwrite` operation in iceberg. In other words, for every update of a datafile, we will `delete` the previous version datafile and `add` the newly created datafile to the iceberg table.
## Need Further Investigations:
1. May need some way to handle `Hard Deletes`
2. We may take advantage of column_stats stored in Hudi's metadata table rather than using FileIO to extract metrics from datafiles.
## Dependency Issue
1. Hudi suggests `Java 8` only, may need further discussion on how to integrate hudi dependency to the iceberg project (may be limit the module to be compiled only for Java 8)
2. `hudi-common` has [dependency conflict](https://github.com/apache/hudi/issues/7409) with the `hudi-spark-bundle`, which is intended to be used for integration test. May need to find alternative way to construct tests later
If you have some comments or suggestions on how to convert hudi to iceberg, please feel free to share them here. Thank you in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on pull request #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-1426338304
Souds Good! Thank you for your suggestions. After the last community sync, I re-investigate the whole process of this demo and realize that there is no proper way to guarantee the correct order of timeline replay.
In the new proposal, the expected migration guarantee is that we always migrate the table at the state of lastest COMPACTION. In other words, we only include the most updated base file in each file group of the hudi table. In this way, users can choose to do compaction before the migration if they want the most updated table or do nothing if they do not want to include newly arrived data.
I will start to investigate the proper way to implement this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on pull request #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-1422908119
[Curiosity] Which one is preferred for hudi-related variable names: `hudixxxx` or `hoodiexxx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-1422911118
> [Curiosity] Which one is preferred for hudi-related variable names: hudixxxx or hoodiexxx
Hudi should be, Hoodie was the name used before the official project name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-2307946373
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-2345009733
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #6642:
URL: https://github.com/apache/iceberg/pull/6642#issuecomment-1423178313
Took a brief look, overall I agree with what the community discussion led to, replaying the timeline is cool but Hudi concurrent transaction has awkward behavior and we cannot guarantee it is always correct.
Instead, we can ask user to always compact before migration, such that we only need to offer the ability to migrate the latest compacted table in Hudi.
When new data come, the user can rerun compaction, and then rerun this migration action, and the action will fully replace the previous version of the migrated Iceberg table.
This approach will work for both CoW and MoR tables, and we can add timeline replay as follow up feature if necessary.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6642: WIP: Support Snapshot Copy-On-Write Hudi Table to Iceberg Table
URL: https://github.com/apache/iceberg/pull/6642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org