You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "yongzhi.shao (Jira)" <ji...@apache.org> on 2024/12/13 15:19:00 UTC
[jira] [Comment Edited] (HIVE-28373) Iceberg: Refactor the code of HadoopTableOptions
[ https://issues.apache.org/jira/browse/HIVE-28373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905524#comment-17905524 ]
yongzhi.shao edited comment on HIVE-28373 at 12/13/24 3:18 PM:
---------------------------------------------------------------
I saw this email, and indeed we can optimize some of the logic. However, in reality, I would prefer to reimplement a catalog based on the file system, as I mentioned in HIVE-28621.
In fact, if we are only managing a single data table, we only need the file system to provide atomic-write (the file becomes visible to all clients immediately after successful write) + limited list operations, without relying on IF-MATCH/atomic-rename and other file system-specific operations.
In this way, we can implement reliable catalog management on any common file system in a generic way.There is no need to determine the type of filesystem the user is using in the code.
was (Author: lisoda):
I saw this email, and indeed we can optimize some of the logic. However, in reality, I would prefer to reimplement a catalog based on the file system, as I mentioned in [HIVE-28621].
In fact, if we are only managing a single data table, we only need the file system to provide atomic-write (the file becomes visible to all clients immediately after successful write) + limited list operations, without relying on IF-MATCH/atomic-rename and other file system-specific operations.
> Iceberg: Refactor the code of HadoopTableOptions
> ------------------------------------------------
>
> Key: HIVE-28373
> URL: https://issues.apache.org/jira/browse/HIVE-28373
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
> Affects Versions: 4.0.0
> Reporter: yongzhi.shao
> Assignee: yongzhi.shao
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Since there are a lot of problems with hadoop_catalog, we submitted the following PR to the iceberg community:
> [core:Refactor the code of HadoopTableOptions by BsoBird · Pull Request #10623 · apache/iceberg (github.com)|https://github.com/apache/iceberg/pull/10623]
> With this PR, we can implement atomic operations based on hadoopcatalog.
> But this PR is not accepted by the iceberg community.And it seems that the iceberg community is trying to remove support for hadoopcatalog(only for testing).
> Since hive itself supports a number of features based on the hadoop_catalog table, can we merge this patch in hive?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)