You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "yongzhi.shao (Jira)" <ji...@apache.org> on 2024/12/13 15:19:00 UTC

[jira] [Comment Edited] (HIVE-28373) Iceberg: Refactor the code of HadoopTableOptions

    [ https://issues.apache.org/jira/browse/HIVE-28373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905524#comment-17905524 ] 

yongzhi.shao edited comment on HIVE-28373 at 12/13/24 3:18 PM:
---------------------------------------------------------------

I saw this email, and indeed we can optimize some of the logic. However, in reality, I would prefer to reimplement a catalog based on the file system, as I mentioned in HIVE-28621.

In fact, if we are only managing a single data table, we only need the file system to provide atomic-write (the file becomes visible to all clients immediately after successful write) + limited list operations, without relying on IF-MATCH/atomic-rename and other file system-specific operations.

In this way, we can implement reliable catalog management on any common file system in a generic way.There is no need to determine the type of filesystem the user is using in the code.


was (Author: lisoda):
I saw this email, and indeed we can optimize some of the logic. However, in reality, I would prefer to reimplement a catalog based on the file system, as I mentioned in [HIVE-28621].

In fact, if we are only managing a single data table, we only need the file system to provide atomic-write (the file becomes visible to all clients immediately after successful write) + limited list operations, without relying on IF-MATCH/atomic-rename and other file system-specific operations.

> Iceberg: Refactor the code of HadoopTableOptions
> ------------------------------------------------
>
>                 Key: HIVE-28373
>                 URL: https://issues.apache.org/jira/browse/HIVE-28373
>             Project: Hive
>          Issue Type: Improvement
>          Components: Iceberg integration
>    Affects Versions: 4.0.0
>            Reporter: yongzhi.shao
>            Assignee: yongzhi.shao
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> Since there are a lot of problems with hadoop_catalog, we submitted the following PR to the iceberg community: 
> [core:Refactor the code of HadoopTableOptions by BsoBird · Pull Request #10623 · apache/iceberg (github.com)|https://github.com/apache/iceberg/pull/10623]
> With this PR, we can implement atomic operations based on hadoopcatalog.
> But this PR is not accepted by the iceberg community.And it seems that the iceberg community is trying to remove support for hadoopcatalog(only for testing).
> Since hive itself supports a number of features based on the hadoop_catalog table, can we merge this patch in hive?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)