You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2025/03/24 04:19:26 UTC
[I] HDFS support in iceberg-rust [iceberg-rust]
Xuanwo opened a new issue, #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130
There are several ways to access HDFS in remote areas:
- libhdfs + JNI: needs java runtime and libhdfs
- WebHDFS: http based, may have performance issues
- [HDFS Native](https://github.com/Kimahriman/hdfs-native): rust native but not well adopted.
OpenDAL supports all these methods. However, from Iceberg's perspective, it's better to expose a single `hdfs` storage option and allow users to choose the method via feature flags to avoid confusion. This is because we write service names in our metadata files, and all implementations must be able to read `hdfs://host:port/files`.
So my current plan is to introduce `webhdfs` and `hdfs_native` support in iceberg-rust. They will hide under `storage-webhdfs` and `storage-hdfs-native` flag:
Users can choose which implementations to use:
```rust
fn parse_scheme(scheme: &str) -> crate::Result<Scheme> {
match scheme {
"memory" => Ok(Scheme::Memory),
"file" | "" => Ok(Scheme::Fs),
"s3" | "s3a" => Ok(Scheme::S3),
"gs" | "gcs" => Ok(Scheme::Gcs),
#[cfg(feature = "storage-webhdfs")]
"hdfs" => Ok(Scheme::Webhdfs),
#[cfg(feature = "storage-hdfs-native")]
"hdfs" => Ok(Scheme::HdfsNative),
s => Ok(s.parse::<Scheme>()?),
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-5240498147
Don't remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2750036582
> 1. It's mainly compile time support or runtime support for webhdfs
What's the difference about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2749860155
cc @liurenjie1024, let's discuss here. I'm also fine to add a new services called `webhdfs` here. But the location returned by hive will still be `hdfs://host:port/abc`. It will be a bit confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2795707391
Does iceberg rust support hadoop catalog?
I only see `s3tables` in the code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2796176496
> Does iceberg rust support hadoop catalog? I only see `s3tables` in the code base.
Hi, @mingmwang There is no support for hadoop catalog, but there is [`StaticTable`](https://github.com/apache/iceberg-rust/blob/6e07faacd7734886718ce544e40599eb2ce939e3/crates/iceberg/src/table.rs#L250) which allows you to query tables without relying on a catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-6350449044
The nice follow up would be supporting `libhdfs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2749995036
> > Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?
>
> Yep. There is no iceberg impls will write `webhdfs://xxx`.
Seems there are several problems needs to resolve here:
1. Should we add webhdfs support as feature? It's mainly compile time support or runtime support for webhdfs
2. Should we treat webhdfs/hdfs as same storage or different service?
Is there other problems to discuss?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2761389382
Per yesterday's discussion, we reached consensus on following points:
1. We should treat webhdfs as a storage different hdfs given their configuration would be different.
2. We should include both libhdfs and hdfsrs in iceberg-rust and allow user to choose at runtime. Libhdfs is still much more mature than hdfsrs so in serious hadoop deployment it's still useful. Hdfsrs allows user to run without relying on, so we think it's still valueful in some use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-5238179825
@liurenjie1024
"Hadoop is still a huge thing " ... still huge and used widely
This is main blocker in datafusion-comet to read Natively HDFS based Iceberg tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2759068343
Sorry that I had to leave early today at the sync, it was a great discussion. Both in Java and PyIceberg there is a FileIO property that you can set to force a specific FileIO: https://py.iceberg.apache.org/configuration/#fileio. This is slightly different since all the implementations come from OpenDAL, but might be an option to consider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-4493465000
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3556512090
> Any plans to resurrect this feature? Hadoop is still a huge thing and it's confusing we can't access Iceberg tables backed by HDFS declaring it's supported in [docs](https://iceberg.apache.org/status/#file-io)
About hdfs there are two things:
1. FileIO support for hdfs. I think this is in the roadmap.
2. Catalog support for hdfs. This is out of scope and deprecated by community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2749945025
> Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?
Yep.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2757068697
> Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?
Yep, I intentionally leaving it out for now since I don't want to play with java runtime for now. And I'm guessing adding it doesn't need another approach. We can just follow the same way to handle it.
> I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.
Oh, that's something new to me. We just need tp map `viewfs` to `hdfs`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3283633189
而且OpenDAL套路有点深，问题很难triage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "oven-yang (via GitHub)" <gi...@apache.org>.
oven-yang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2757044862
Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2749898144
> cc [@liurenjie1024](https://github.com/liurenjie1024), let's discuss here.
>
> I'm also fine to add a new services called `webhdfs` here. But the location returned by hive will still be `hdfs://host:port/abc`. It will be a bit confusing.
Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3275100664
Analyzing current status of HDFS support...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3283550391
## 🔄 HDFS支持实施状态更新 (2025年9月12日)
### ✅ 技术分析完成
经过全面分析，确认以下技术要点：
1. **OpenDAL 0.54.0 完全支持**：确认支持三种HDFS服务
- `Hdfs` (libhdfs + JNI)
- `WebHdfs` (HTTP REST API)
- `HdfsNative` (纯Rust实现)
2. **存储架构已就绪**：当前Storage枚举架构完全支持扩展HDFS变体
- 现有模式：Memory、LocalFs、S3、GCS、OSS、Azdls
- 可直接添加三个HDFS变体，遵循相同的配置和操作符创建模式
3. **社区共识明确**：基于2025年3月讨论达成的一致方向
- 三个独立Storage变体
- 运行时选择机制（通过feature flags）
- 支持viewfs://协议映射
### 📋 实施计划
**优先级顺序**：WebHdfs > Hdfs > HdfsNative
**技术实施步骤**：
1. 添加feature flags：`storage-hdfs`、`storage-webhdfs`、`storage-hdfs-native`
2. 在Storage枚举中添加三个HDFS变体
3. 实现配置解析和操作符创建函数
4. 在parse_scheme中添加协议映射（hdfs://、webhdfs://、viewfs://）
5. 设计运行时选择机制
### ⚠️ 关键限制说明
- **WebHDFS NameNode HA限制**：如@manuzhang指出，WebHDFS不支持自动故障切换，需在文档中突出说明
- **libhdfs运行时要求**：需要Java环境支持
### 🔍 PR #1450 状态确认
经检查，PR #1450 当前状态：
- 状态：CONFLICTING（合并冲突）
- 最后更新：2025年6月18日
- 技术方向：专注于Hadoop catalog实现，而非Storage层面
该PR方向与当前Storage层面需求不同，需要与@awol2005ex进一步沟通确认后续处理方案。
### 🎯 下一步行动
准备开始具体实施，将基于现有Storage架构模式直接实现HDFS支持，无需等待存在冲突的PR。
估计实施时间：2-3周完成核心功能
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2750146226
> > > 1. It's mainly compile time support or runtime support for webhdfs
> >
> >
> > What's the difference about this?
>
> Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.
I see, you mean that we have a single Storage `Hdfs` but provides a config like `use_webhdfs` for users to switch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2750166032
I see. We have three approachs so far:
1. Support `storage-webhdfs` and `storage-hdfs` but only exposes as `hdfs` to users
Users are required to use feature flags for configuration.
2. Have only `storage-hdfs` but allow users to pick up impls like `webhdfs` or `hdfs` at runtime by config.
Provides more control for users, but it may be a bit confusing when feature flags don't match the configuration.
3. Exposes `storage-webhdfs` and `storage-hdfs` seperately
It doesn't work for webhdfs since all tables will use `hdfs://abc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2761567656
AFAIK, `webhdfs` doesn't support HDFS NameNode HA. When fail over happens, it won't switch to the new active NameNode automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "EugeneYushin (via GitHub)" <gi...@apache.org>.
EugeneYushin commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3553181545
Any plans to resurrect this feature? Hadoop is still a huge thing and it's confusing we can't access Iceberg tables backed by HDFS declaring it's supported in [docs](https://iceberg.apache.org/status/#file-io)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2750144000
> > 1. It's mainly compile time support or runtime support for webhdfs
>
> What's the difference about this?
Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "oven-yang (via GitHub)" <gi...@apache.org>.
oven-yang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2757061211
I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "oven-yang (via GitHub)" <gi...@apache.org>.
oven-yang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2757182202
> > Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?
>
> Yep, I intentionally leaving it out for now since I don't want to play with java runtime for now. And I'm guessing adding it doesn't need another approach. We can just follow the same way to handle it.
I actually tested accessing HDFS using libhdfs + JNI myself, and it worked! The main issue I ran into was linking the JVM library.
It’d be awesome if support for this could be added here eventually.
> > I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.
>
> Oh, that's something new to me. We just need tp map `viewfs` to `hdfs`, right?
To clarify, I ran my test using libhdfs + JNI, and I’m not entirely sure if `viewfs` is supported in the other two approaches.
In my experience with libhdfs + JNI, `viewfs` is handled the same way as `hdfs`, with libhdfs taking care of the differences under the hood. That said, it seems like we’d only need to explicitly pass the scheme to OpenDAL for it to work smoothly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-2750151741
> > Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.
>
> I see, you mean that we have a single Storage `Hdfs` but provides a config like `use_webhdfs` for users to switch?
Yes, but I'm not sure it's the best approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3283631453
OpenDAL的HDFS写路径问题挺多的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3283595371
@Xuanwo Are you doing some experiment with AI summary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] HDFS support in iceberg-rust [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1130:
URL: https://github.com/apache/iceberg-rust/issues/1130#issuecomment-3284300830
> [@Xuanwo](https://github.com/Xuanwo) Are you doing some experiment with AI summary?
Oh, sorry about that. My AI summary bot accidentally posted the comments publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org