You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mixermt (via GitHub)" <gi...@apache.org> on 2026/08/30 06:58:25 UTC
[PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
mixermt opened a new pull request, #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111
## Which issue does this PR close?
- Closes #2440.
- Part of #1130.
This revives #1131's successor #2441 by @jordepic (closed by the stale bot after a first review round), rebased onto the current `Storage`-trait layout and updated for opendal 0.58, where `services-hdfs-native` changed behavior in ways that required design changes (details below).
## What changes are included in this PR?
Adds an `opendal-hdfs-native` cargo feature to `iceberg-storage-opendal`, with `OpenDalStorageFactory::Hdfs` / `OpenDalStorage::Hdfs` variants and `hdfs://` routing in `OpenDalResolvingStorage`. The backend uses OpenDAL's `services-hdfs-native` (pure-Rust HDFS RPC via `hdfs-native` — no JNI/libhdfs). The feature is experimental and not part of `opendal-all`, matching `opendal-oss`/`opendal-azdls`.
NameNode resolution (differs from #2441, forced by opendal 0.58 where `name_node` is mandatory and its comma-split list is the HA mechanism):
1. `hdfs.name-node` property when set — a single endpoint or a comma-separated list for HA failover (new `HdfsConfig` + constants in `iceberg::io`, following the existing per-backend config convention).
2. Otherwise the path authority (`hdfs://host:port/path`).
3. Authority-less paths without the property are rejected with a pointed error (opendal 0.58 can no longer defer to `fs.defaultFS`).
`hadoop.`-prefixed properties pass through to the HDFS client config, overriding `$HADOOP_CONF_DIR` values (mirroring the `hadoop.` catalog-property convention of the Java integrations); `hdfs-native` still loads `core-site.xml`/`hdfs-site.xml` from `$HADOOP_CONF_DIR`/`$HADOOP_HOME` for everything else, and Kerberos works via `libgssapi_krb5` (runtime dlopen). Operators are cached per effective NameNode since each holds live RPC connections.
Relative paths are returned opendal-style without a leading `/` — `opendal::Deleter::delete` (used by `delete_stream`) rejects leading slashes, which an integration test caught.
Test infrastructure: single-node HDFS docker fixture (`apache/hadoop:3.5.0`, host networking — required because `hdfs-native` dials DataNodes by their registered IP, unroutable on a bridge). The DataNode healthcheck gates on NameNode registration so `--wait` means writable. Tests are `#[ignore]`d (host networking is Linux-only); CI opts in via `cargo nextest --run-ignored=only -E 'test(file_io_hdfs)'`.
## Are these changes tested?
- 24 unit tests across config parsing, path parsing (authority/port/authority-less/wrong-scheme), NameNode precedence, operator caching, relativize, and scheme resolution; 3 config tests in `iceberg::io`.
- 12 integration tests against the docker fixture covering exists/read/write/overwrite/metadata/range reads/streaming writes/delete/delete_prefix/delete_stream (+empty), resolving storage over `hdfs://`, and the HA flow (logical authority in the path + `hdfs.name-node` property). All 12 verified green from a cold cluster locally; the suite runs in CI on Linux.
- `make check`, full-workspace `--all-features` lib tests, and the existing s3/gcs/resolving integration suites pass; `public-api.txt` regenerated for both crates.
## AI Disclosure
Developed with AI assistance (Claude Code): drafting code/tests/fixtures starting from #2441, and cross-checking the design against the vendored opendal 0.58.1 / hdfs-native 0.14.5 sources. I reviewed the implementation and ran all verification locally. Areas worth reviewer attention: the NameNode-resolution semantics above (opendal's synthetic-nameservice behavior constrains what `hdfs://<nameservice>` paths can do without the property), and the Windows `--all-features` build of `hdfs-native`, which I could only verify via CI.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3932278311
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use iceberg_property_macro::Properties;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+// No in-crate consumer yet: `iceberg-storage-opendal` parses the raw
+// properties itself and only shares the key constants above.
+#[allow(dead_code)]
Review Comment:
No, we don't need this, make this `HdfsConfig` pub
##########
crates/storage/opendal/src/hdfs.rs:
##########
Review Comment:
Rename this to hdfs_native.
##########
crates/storage/opendal/src/hdfs.rs:
##########
@@ -0,0 +1,349 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
Review Comment:
All functions should have prefix hdfs_native
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use iceberg_property_macro::Properties;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+// No in-crate consumer yet: `iceberg-storage-opendal` parses the raw
+// properties itself and only shares the key constants above.
+#[allow(dead_code)]
Review Comment:
Seems this have never been used? I think we could remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4169088910
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -460,6 +499,11 @@ impl OpenDalStorage {
))
}
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ OpenDalStorage::HdfsNative { .. } => {
+ let (_, relative_path) = hdfs_native_parse_path(path)?;
Review Comment:
`relativize_path` only calls `hdfs_native_parse_path`, so `hdfs:///a/b` returns `Ok("a/b")` here while `create_operator` on the same unconfigured path errors — the two disagree on whether an authority-less path is usable, and every other backend fails fast in both. I'd route this through `hdfs_native_effective_name_node` so a path that can't resolve a NameNode errors the same way on both.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
Review Comment:
`url.host_str()` strips the brackets off an IPv6 literal, so `hdfs://[::1]:8020` reconstructs here as `hdfs://::1:8020` — not a parseable URL, and both `url` and `hdfs-native` reject it. Any IPv6 NameNode then fails every operator build and cache lookup.
`url.host()` keeps the bracketed form via its `Display`:
```rust
let name_node = url.host().filter(|h| !matches!(h, url::Host::Domain(""))).map(|host| {
url.port().map(|port| format!("hdfs://{host}:{port}")).unwrap_or_else(|| format!("hdfs://{host}"))
});
```
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +261,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: HdfsNativeOperatorCache,
Review Comment:
@comphead already flagged this — seconding it. `operators` as a `pub` field drags `HdfsNativeOperatorCache` into the crate's public API (it's in `public-api.txt` now), which no other variant does — they expose only `config`. That locks a connection-pooling detail into semver and lets a caller build the variant with an isolated cache, silently losing reuse. I'd wrap the state in a newtype, `HdfsNative(Arc<HdfsNativeState>)` with private internals (mirrors `Memory(Operator)`), and drop the `pub use`. While we're here, a line documenting the cache lifecycle — one entry per effective NameNode, lives with the storage instance, no eviction — would help too.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority — plus the relative
+/// path. Both the operator cache and `delete_stream` batching key on this,
+/// so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // Built outside the lock: the build reads the Hadoop XML config
+ // synchronously. A racing first caller may build too; the loser is
+ // dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ Ok((operators.insert(name_node, op)?, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, so paths that resolve to different operators never share a
+/// deleter. Unresolvable paths key on themselves (as `hf_batch_key` does);
+/// `create_operator` then reports the real error.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
+ hdfs_native_effective_name_node(config, path)
+ .map(|(name_node, _)| name_node)
+ .unwrap_or_else(|_| path.to_string())
+}
+
+/// Build a new OpenDAL [`Operator`]: OpenDAL splits `name_node` on commas
+/// into a synthetic HA name service; `$HADOOP_CONF_DIR` XML still merges in.
+fn hdfs_native_operator_build(config: &HdfsNativeConfig, name_node: &str) -> Result<Operator> {
+ let mut cfg = config.clone();
+ cfg.name_node = Some(name_node.to_string());
+ Operator::from_config(cfg).map_err(from_opendal_error)
Review Comment:
`Operator::from_config` reads the Hadoop XML synchronously, and this runs on the tokio executor thread (sync fn reached from the async `Storage` methods) with no `spawn_blocking`. @comphead's "blocking under the write lock" point is handled — the build is outside the lock now — but the cold build still blocks the executor thread, so first-access-per-NameNode under load stalls other tasks. I'd make `hdfs_native_create_operator` async and wrap the build in `spawn_blocking`, or document it as a known experimental constraint with a tracking issue.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -135,6 +144,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
@blackmwk already raised this and I agree — every other variant is named for the storage system (`Gcs`, `Oss`, `Azdls`, `Hf`), so `HdfsNative` is the odd one out leaking the opendal implementation choice. I'd name the public variant `Hdfs` and keep `services-hdfs-native` as the internal/feature detail.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
Review Comment:
A bare `hdfs.name-node = "namenode:8020"` (no `hdfs://`) is accepted and passed straight through, and `hdfs-native` then fails with an opaque error rather than something that points at the misconfigured property. I'd validate the value starts with `hdfs://` here and return `DataInvalid` otherwise.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
Review Comment:
PyIceberg already has a de-facto convention here — `hdfs.host` + `hdfs.port` (plus `hdfs.user` / `hdfs.kerberos_ticket`) — and this introduces `hdfs.name-node` as a single combined key, so catalog properties carried over from PyIceberg get silently ignored. It's not a metadata-interop issue (these keys aren't stored in table metadata), but it's a real usability gap. At minimum I'd note in the doc-comment that this key is iceberg-rust/hdfs-native-specific and differs from PyIceberg; accepting `hdfs.host`+`hdfs.port` as a fallback would be better still.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority — plus the relative
+/// path. Both the operator cache and `delete_stream` batching key on this,
+/// so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // Built outside the lock: the build reads the Hadoop XML config
+ // synchronously. A racing first caller may build too; the loser is
+ // dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ Ok((operators.insert(name_node, op)?, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, so paths that resolve to different operators never share a
+/// deleter. Unresolvable paths key on themselves (as `hf_batch_key` does);
+/// `create_operator` then reports the real error.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
+ hdfs_native_effective_name_node(config, path)
+ .map(|(name_node, _)| name_node)
+ .unwrap_or_else(|_| path.to_string())
+}
+
+/// Build a new OpenDAL [`Operator`]: OpenDAL splits `name_node` on commas
+/// into a synthetic HA name service; `$HADOOP_CONF_DIR` XML still merges in.
+fn hdfs_native_operator_build(config: &HdfsNativeConfig, name_node: &str) -> Result<Operator> {
+ let mut cfg = config.clone();
+ cfg.name_node = Some(name_node.to_string());
+ Operator::from_config(cfg).map_err(from_opendal_error)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_hdfs_native_config_parse_name_node_and_options() {
+ let props = HashMap::from([
+ (
+ HDFS_NAME_NODE.to_string(),
+ "hdfs://nn1:8020,hdfs://nn2:8020".to_string(),
+ ),
+ (
+ "hadoop.dfs.client.failover.random.order".to_string(),
+ "true".to_string(),
+ ),
+ ("unrelated.key".to_string(), "ignored".to_string()),
+ ]);
+
+ let cfg = hdfs_native_config_parse(props).unwrap();
+
+ assert_eq!(
+ cfg.name_node.as_deref(),
+ Some("hdfs://nn1:8020,hdfs://nn2:8020")
+ );
+ let options = cfg.options.unwrap();
+ assert_eq!(
+ options.get("dfs.client.failover.random.order"),
+ Some(&"true".to_string())
+ );
+ assert!(!options.contains_key("unrelated.key"));
+ }
+
+ #[test]
+ fn test_hdfs_native_config_parse_empty() {
+ let cfg = hdfs_native_config_parse(HashMap::new()).unwrap();
+
+ assert_eq!(cfg.name_node, None);
+ assert_eq!(cfg.options, None);
+ }
+
+ #[test]
+ fn test_hdfs_native_config_parse_normalizes_name_node() {
+ let parse = |value: &str| {
+ hdfs_native_config_parse(HashMap::from([(
+ HDFS_NAME_NODE.to_string(),
+ value.to_string(),
+ )]))
+ .unwrap()
+ .name_node
+ };
+
+ // Empty must not shadow the path-authority fallback.
+ assert_eq!(parse(""), None);
+ assert_eq!(parse(" "), None);
+ // Trailing `/` would otherwise yield a second cache entry for one cluster.
+ assert_eq!(
+ parse(" hdfs://nn:8020/ ").as_deref(),
+ Some("hdfs://nn:8020")
+ );
+ }
+
+ #[test]
+ fn test_hdfs_native_effective_name_node_precedence() {
+ let configured = hdfs_native_config_parse(HashMap::from([(
+ HDFS_NAME_NODE.to_string(),
+ "hdfs://nn1:8020,hdfs://nn2:8020".to_string(),
+ )]))
+ .unwrap();
+ let unconfigured = HdfsNativeConfig::default();
+
+ // Configured wins over the authority, including for authority-less paths.
+ for path in ["hdfs://ns-a/x", "hdfs:///y"] {
+ let (nn, _) = hdfs_native_effective_name_node(&configured, path).unwrap();
+ assert_eq!(nn, "hdfs://nn1:8020,hdfs://nn2:8020");
+ }
+ // Otherwise the authority, including its port.
+ let (nn, rel) =
+ hdfs_native_effective_name_node(&unconfigured, "hdfs://nn:9000/a/b").unwrap();
+ assert_eq!((nn.as_str(), rel), ("hdfs://nn:9000", "a/b"));
+ // Neither: a pointed error.
+ let err = hdfs_native_effective_name_node(&unconfigured, "hdfs:///a").unwrap_err();
+ assert!(err.to_string().contains(HDFS_NAME_NODE));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_and_rel() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1/a/b").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "a/b");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_and_port() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nn:8020/foo").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nn:8020"));
+ assert_eq!(rel, "foo");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_no_path() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_trailing_slash() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1/").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_authority_less_returns_none() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs:///a/b").unwrap();
+
+ assert_eq!(nn, None);
+ assert_eq!(rel, "a/b");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_wrong_scheme_errors() {
+ let err = hdfs_native_parse_path("file:///tmp/x").unwrap_err();
+
+ assert!(err.to_string().contains("expected scheme `hdfs://`"));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_invalid_url_errors() {
+ let err = hdfs_native_parse_path("not-a-url").unwrap_err();
+
+ assert!(err.to_string().contains("Invalid hdfs path"));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_non_hierarchical_errors() {
+ // `hdfs:x` parses as a valid non-hierarchical URL; it must be
+ // rejected rather than panic on slicing.
+ for path in ["hdfs:x", "hdfs:/x", "hdfs:"] {
+ let err = hdfs_native_parse_path(path).unwrap_err();
+ assert!(err.to_string().contains("expected scheme `hdfs://`"));
+ }
+ }
+
+ #[test]
+ fn test_hdfs_native_batch_key_distinguishes_ports() {
+ let config = HdfsNativeConfig::default();
+
+ assert_eq!(
+ hdfs_native_batch_key(&config, "hdfs://namenode:8020/a"),
+ "hdfs://namenode:8020"
+ );
+ assert_eq!(
+ hdfs_native_batch_key(&config, "hdfs://namenode:9000/b"),
+ "hdfs://namenode:9000"
+ );
+ }
+
+ #[test]
+ fn test_hdfs_native_batch_key_invalid_path_keys_on_itself() {
+ let config = HdfsNativeConfig::default();
+
+ // Unresolvable paths must not collapse onto a shared "" key.
+ assert_eq!(hdfs_native_batch_key(&config, "not-a-url"), "not-a-url");
+ assert_eq!(hdfs_native_batch_key(&config, "hdfs:///a"), "hdfs:///a");
+ }
+
+ #[test]
Review Comment:
These are plain `#[test]`s with no runtime, relying on `Operator::from_config` being runtime-free today. If a future `hdfs-native` builds the `Client` eagerly (it captures `Handle::try_current()` to spawn tasks), these panic with "no reactor running" — the same eager-construction path behind @comphead's runtime-pinning question. I'd switch them to `#[tokio::test]` and document whether `from_config` is actually runtime-free, so the contract is pinned rather than incidental.
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,315 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the HDFS fixture in `dev/docker-compose.yaml` and are
+//! skipped when `ICEBERG_TEST_HDFS_ENDPOINT` is not set. The fixture uses
+//! host networking (Linux, or a Docker runtime that supports it), so it sits
+//! behind a compose profile:
+//!
+//! ```text
+//! COMPOSE_PROFILES=hdfs make docker-up
+//! ICEBERG_TEST_HDFS_ENDPOINT=hdfs://localhost:8020 cargo test -p iceberg-storage-opendal \
+//! --features opendal-hdfs-native --test file_io_hdfs_test
+//! ```
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::sync::Arc;
+
+ use bytes::Bytes;
+ use futures::StreamExt;
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::{OpenDalResolvingStorageFactory, OpenDalStorageFactory};
+ use iceberg_test_utils::{
+ ENV_HDFS_ENDPOINT, get_hdfs_endpoint, normalize_test_name_with_parts, set_up,
+ };
+
+ /// Skips the calling test unless the HDFS fixture endpoint is configured;
+ /// an unset *or* empty variable means "not provided" (see the HF tests).
+ macro_rules! require_hdfs {
+ () => {
+ match std::env::var(ENV_HDFS_ENDPOINT) {
+ Ok(v) if !v.is_empty() => {}
+ _ => {
+ eprintln!("Skipping HDFS test: {} not set", ENV_HDFS_ENDPOINT);
+ return;
+ }
+ }
+ };
+ }
+
+ fn get_file_io() -> FileIO {
+ set_up();
+ FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative)).build()
+ }
+
+ fn test_path(suffix: &str) -> String {
+ format!(
+ "{}/{}",
+ get_hdfs_endpoint(),
+ normalize_test_name_with_parts!(suffix)
+ )
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_exists() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let absent = test_path("test_file_io_hdfs_exists_absent");
+ assert!(!file_io.exists(&absent).await.unwrap());
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_write_and_read() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_write_and_read");
+ let _ = file_io.delete(&path).await;
+
+ let output = file_io.new_output(&path).unwrap();
+ output
+ .write(Bytes::from_static(b"hello hdfs"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ let input = file_io.new_input(&path).unwrap();
+ assert_eq!(
+ input.read().await.unwrap(),
+ Bytes::from_static(b"hello hdfs")
+ );
+ }
+
+ /// The HA flow: table locations carry a logical authority while
+ /// `hdfs.name-node` carries the (comma-separated) endpoints; it wins.
+ #[tokio::test]
+ async fn test_file_io_hdfs_configured_name_node() {
+ require_hdfs!();
+ set_up();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, get_hdfs_endpoint())
+ .build();
+
+ // The path authority is a logical name; the configured NameNode wins.
+ let path = format!(
+ "hdfs://logical-nameservice/{}",
+ normalize_test_name_with_parts!("test_file_io_hdfs_configured_name_node")
+ );
+ let _ = file_io.delete(&path).await;
+
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(b"via configured name node"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"via configured name node")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_overwrite() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_overwrite");
+ let _ = file_io.delete(&path).await;
+
+ for content in [b"first".as_slice(), b"second, longer".as_slice()] {
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(content))
+ .await
+ .unwrap();
+ }
+
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"second, longer")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream() {
Review Comment:
The batch-key-by-effective-NameNode logic is the one real behavioral difference from the other backends, and it's never exercised end-to-end with two distinct NameNodes — this test uses one. I'd add a case with paths across two effective NameNodes (on the single-node fixture, `localhost` vs `127.0.0.1` gives two keys) asserting both groups delete. A concurrent cache-insert unit test racing two callers for the same NameNode (asserting `len() == 1`) would also lock in the `or_insert` race handling, and that one needs no real HDFS.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
Review Comment:
`trim_end_matches('/')` only trims the end of the whole string, so for a comma-separated HA list like `hdfs://nn1:8020/,hdfs://nn2:8020/` the first entry keeps its slash and we cache two operators for one cluster. I'd split on `,` and trim each entry before rejoining.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
Review Comment:
The doc doesn't quite say the `hadoop.` prefix is stripped before forwarding. Worth spelling out — something like "property `hadoop.<key>` is forwarded to the HDFS client config as `<key>`".
##########
crates/test_utils/src/lib.rs:
##########
@@ -47,13 +47,15 @@ mod common {
pub const ENV_HMS_ENDPOINT: &str = "ICEBERG_TEST_HMS_ENDPOINT";
pub const ENV_GLUE_ENDPOINT: &str = "ICEBERG_TEST_GLUE_ENDPOINT";
pub const ENV_GCS_ENDPOINT: &str = "ICEBERG_TEST_GCS_ENDPOINT";
+ pub const ENV_HDFS_ENDPOINT: &str = "ICEBERG_TEST_HDFS_ENDPOINT";
// Default ports matching dev/docker-compose.yaml
pub const DEFAULT_OBJECT_STORE_PORT: u16 = 9000;
pub const DEFAULT_REST_CATALOG_PORT: u16 = 8181;
pub const DEFAULT_HMS_PORT: u16 = 9083;
pub const DEFAULT_GLUE_PORT: u16 = 5001;
pub const DEFAULT_GCS_PORT: u16 = 4443;
+ pub const DEFAULT_HDFS_NN_PORT: u16 = 8020;
Review Comment:
The siblings are all `DEFAULT_<SERVICE>_PORT` — this one slips in an `_NN_` infix. I'd rename to `DEFAULT_HDFS_PORT` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5712429405
Thanks @comphead for the thorough review — all five items are addressed in d17ae38 and answered inline. The `security_audit` failure was RUSTSEC-2026-0285 in `rustls 0.23.41`, which is main's current pin (AWS SDK chain, unrelated to hdfs-native; tracked in #3222). Bumped to 0.23.45 in b8cc860, the same fix #3165 applied. The minor simplification notes you held back are welcome — please send them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4231835012
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
Review Comment:
**Nit (manual slicing):** `key[nameservice_prefix.len()..]` relies on a `starts_with` check made in a separate statement above. Using `key.strip_prefix(&nameservice_prefix)` here, or collecting `(nameservice, value)` pairs from `strip_prefix` in the first pass, makes it panic-free by construction and avoids the key clones and the `remove(&key).unwrap_or_default()`. The same applies to `s[..i]`, `s[start..]` and `s[start + at..]` in `hdfs_native_redact` (L252 to 258) and `after_scheme[i..]` in `hdfs_native_parse_path` (L300), which could use `split_once` or `strip_prefix`. The indices come from `find`, so I do not see a panic today.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
Review Comment:
**Should fix (strong types):** following up on my earlier comment here. cheats.rs recommends `struct Charge(f32)` over a bare `f32` and enums over flags, so that illegal states cannot be built. A NameNode is currently a `String` that can be an endpoint, a logical nameservice or a comma-separated list. `hdfs_native_parse_path` returns `Option<String>`, and this function re-parses that string to learn which kind it is. The results are tuples such as `(String, &'a str)`.
A `NameNode` newtype and an `Authority` enum would make the states explicit and remove the re-parse.
The `hdfs_native_` prefix on every free function looks like a workaround for the flat `use hdfs_native::*;` in `lib.rs`. With a resolver type these could be methods and the prefix would not be needed. This part is my inference, not a cheats.rs rule.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -186,6 +194,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
**Question (API stability):** cheats.rs lists "Adding new enum variants" as breaking and suggests an early `#[non_exhaustive]`. `OpenDalStorageFactory` and `OpenDalStorage` do not have it, so a downstream exhaustive `match` stops compiling when a feature-unified build turns on `opendal-hdfs-native`. Earlier backends did the same and the feature is off by default, so I would not block on it. Would it be worth adding `#[non_exhaustive]` to both enums in a separate PR?
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
Review Comment:
**Nit (iterator chains):** cheats.rs says "Functional style is often easiest to follow, but don't hesitate to use `for` if your `.iter()` chain turns messy." Three chains in this file fit that.
- Here, `remove().map(trim.to_string).filter(!empty).map(parse.map_err(Error::new(..))).transpose()?` nests a fallible closure inside an `Option` chain. The `hdfs.host` lookup above repeats the same `map.filter`. A small helper such as `fn trimmed_nonempty(m: &mut HashMap<String, String>, key: &str) -> Option<String>` plus a `match` on the parse would read better.
- `hdfs_native_nameservice` (L366) chains `split.map.filter.map(|id| options.get.and_then.ok_or_else(..)).collect::<Result<Vec<_>>>()?`. A `for` loop with `?` is simpler.
- `hdfs_native_create_operator` (L538) ends in `.map_err(..)??`. Two separate `?` lines would show which error is the join error and which is the build error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6089145131
@comphead on the review notes: the declared-nameservice test now lists a dead NameNode first, so every call has to fail over through opendal's synthetic nameservice, and keeping only the first entry of the list makes it fail. The Windows `--all-features` build is covered by CI's `build (windows-latest)` job (`cargo build --all-targets --all-features --workspace`), which passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3940146474
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use iceberg_property_macro::Properties;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+// No in-crate consumer yet: `iceberg-storage-opendal` parses the raw
+// properties itself and only shares the key constants above.
+#[allow(dead_code)]
Review Comment:
Removed in a4178fe — `config/hdfs.rs` now carries only the `HDFS_NAME_NODE` / `HDFS_HADOOP_CONF_PREFIX` constants that `iceberg-storage-opendal` consumes.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
Review Comment:
Renamed to `hdfs_native.rs` in a4178fe. For consistency I also renamed the variants `OpenDalStorageFactory::Hdfs` / `OpenDalStorage::Hdfs` → `HdfsNative` (leaves room for a libhdfs-backed variant per #1130) — say so if you'd rather keep `Hdfs`.
##########
crates/storage/opendal/src/hdfs.rs:
##########
@@ -0,0 +1,349 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
Review Comment:
Done in a4178fe: `hdfs_native_config_parse`, `hdfs_native_parse_path`, `hdfs_native_create_operator`, `hdfs_native_batch_key`, `hdfs_native_operator_build`.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
Superseded — the struct is removed entirely in a4178fe (see the thread below).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4174607506
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority — plus the relative
+/// path. Both the operator cache and `delete_stream` batching key on this,
+/// so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // Built outside the lock: the build reads the Hadoop XML config
+ // synchronously. A racing first caller may build too; the loser is
+ // dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ Ok((operators.insert(name_node, op)?, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, so paths that resolve to different operators never share a
+/// deleter. Unresolvable paths key on themselves (as `hf_batch_key` does);
+/// `create_operator` then reports the real error.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
+ hdfs_native_effective_name_node(config, path)
+ .map(|(name_node, _)| name_node)
+ .unwrap_or_else(|_| path.to_string())
+}
+
+/// Build a new OpenDAL [`Operator`]: OpenDAL splits `name_node` on commas
+/// into a synthetic HA name service; `$HADOOP_CONF_DIR` XML still merges in.
+fn hdfs_native_operator_build(config: &HdfsNativeConfig, name_node: &str) -> Result<Operator> {
+ let mut cfg = config.clone();
+ cfg.name_node = Some(name_node.to_string());
+ Operator::from_config(cfg).map_err(from_opendal_error)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_hdfs_native_config_parse_name_node_and_options() {
+ let props = HashMap::from([
+ (
+ HDFS_NAME_NODE.to_string(),
+ "hdfs://nn1:8020,hdfs://nn2:8020".to_string(),
+ ),
+ (
+ "hadoop.dfs.client.failover.random.order".to_string(),
+ "true".to_string(),
+ ),
+ ("unrelated.key".to_string(), "ignored".to_string()),
+ ]);
+
+ let cfg = hdfs_native_config_parse(props).unwrap();
+
+ assert_eq!(
+ cfg.name_node.as_deref(),
+ Some("hdfs://nn1:8020,hdfs://nn2:8020")
+ );
+ let options = cfg.options.unwrap();
+ assert_eq!(
+ options.get("dfs.client.failover.random.order"),
+ Some(&"true".to_string())
+ );
+ assert!(!options.contains_key("unrelated.key"));
+ }
+
+ #[test]
+ fn test_hdfs_native_config_parse_empty() {
+ let cfg = hdfs_native_config_parse(HashMap::new()).unwrap();
+
+ assert_eq!(cfg.name_node, None);
+ assert_eq!(cfg.options, None);
+ }
+
+ #[test]
+ fn test_hdfs_native_config_parse_normalizes_name_node() {
+ let parse = |value: &str| {
+ hdfs_native_config_parse(HashMap::from([(
+ HDFS_NAME_NODE.to_string(),
+ value.to_string(),
+ )]))
+ .unwrap()
+ .name_node
+ };
+
+ // Empty must not shadow the path-authority fallback.
+ assert_eq!(parse(""), None);
+ assert_eq!(parse(" "), None);
+ // Trailing `/` would otherwise yield a second cache entry for one cluster.
+ assert_eq!(
+ parse(" hdfs://nn:8020/ ").as_deref(),
+ Some("hdfs://nn:8020")
+ );
+ }
+
+ #[test]
+ fn test_hdfs_native_effective_name_node_precedence() {
+ let configured = hdfs_native_config_parse(HashMap::from([(
+ HDFS_NAME_NODE.to_string(),
+ "hdfs://nn1:8020,hdfs://nn2:8020".to_string(),
+ )]))
+ .unwrap();
+ let unconfigured = HdfsNativeConfig::default();
+
+ // Configured wins over the authority, including for authority-less paths.
+ for path in ["hdfs://ns-a/x", "hdfs:///y"] {
+ let (nn, _) = hdfs_native_effective_name_node(&configured, path).unwrap();
+ assert_eq!(nn, "hdfs://nn1:8020,hdfs://nn2:8020");
+ }
+ // Otherwise the authority, including its port.
+ let (nn, rel) =
+ hdfs_native_effective_name_node(&unconfigured, "hdfs://nn:9000/a/b").unwrap();
+ assert_eq!((nn.as_str(), rel), ("hdfs://nn:9000", "a/b"));
+ // Neither: a pointed error.
+ let err = hdfs_native_effective_name_node(&unconfigured, "hdfs:///a").unwrap_err();
+ assert!(err.to_string().contains(HDFS_NAME_NODE));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_and_rel() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1/a/b").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "a/b");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_and_port() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nn:8020/foo").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nn:8020"));
+ assert_eq!(rel, "foo");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_no_path() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_with_authority_trailing_slash() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs://nameservice1/").unwrap();
+
+ assert_eq!(nn.as_deref(), Some("hdfs://nameservice1"));
+ assert_eq!(rel, "");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_authority_less_returns_none() {
+ let (nn, rel) = hdfs_native_parse_path("hdfs:///a/b").unwrap();
+
+ assert_eq!(nn, None);
+ assert_eq!(rel, "a/b");
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_wrong_scheme_errors() {
+ let err = hdfs_native_parse_path("file:///tmp/x").unwrap_err();
+
+ assert!(err.to_string().contains("expected scheme `hdfs://`"));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_invalid_url_errors() {
+ let err = hdfs_native_parse_path("not-a-url").unwrap_err();
+
+ assert!(err.to_string().contains("Invalid hdfs path"));
+ }
+
+ #[test]
+ fn test_hdfs_native_parse_path_non_hierarchical_errors() {
+ // `hdfs:x` parses as a valid non-hierarchical URL; it must be
+ // rejected rather than panic on slicing.
+ for path in ["hdfs:x", "hdfs:/x", "hdfs:"] {
+ let err = hdfs_native_parse_path(path).unwrap_err();
+ assert!(err.to_string().contains("expected scheme `hdfs://`"));
+ }
+ }
+
+ #[test]
+ fn test_hdfs_native_batch_key_distinguishes_ports() {
+ let config = HdfsNativeConfig::default();
+
+ assert_eq!(
+ hdfs_native_batch_key(&config, "hdfs://namenode:8020/a"),
+ "hdfs://namenode:8020"
+ );
+ assert_eq!(
+ hdfs_native_batch_key(&config, "hdfs://namenode:9000/b"),
+ "hdfs://namenode:9000"
+ );
+ }
+
+ #[test]
+ fn test_hdfs_native_batch_key_invalid_path_keys_on_itself() {
+ let config = HdfsNativeConfig::default();
+
+ // Unresolvable paths must not collapse onto a shared "" key.
+ assert_eq!(hdfs_native_batch_key(&config, "not-a-url"), "not-a-url");
+ assert_eq!(hdfs_native_batch_key(&config, "hdfs:///a"), "hdfs:///a");
+ }
+
+ #[test]
Review Comment:
Checked this: hdfs-native already builds the client eagerly and resolves the handle at build, falling back to a private runtime when there is none, so a plain `#[test]` cannot hit "no reactor running" (the 59 unit tests run that path). I kept them plain on purpose and said so in a comment: if a build ever starts needing a runtime, these fail loudly, whereas under `#[tokio::test]` they would keep passing. On comphead's pinning point, I verified the real constraint: an operator first used in runtime A panics inside hdfs-native when used from runtime B after A is dropped (it unwraps the join error, `connection.rs:57`, still in 0.14.6). The cache doc now states that a storage must not be reused across runtimes that way. If reuse across runtimes matters to someone, the cache can detect a dead runtime and rebuild; I'd rather do that as a follow-up than widen this PR. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5973354271
Thanks @laskoviymishka, thorough review. All 12 threads are addressed in eeca0d1..22db8e6 (fresh merge of main plus 10 small commits), with a reply on each. Behavior changes: `relativize_path` resolves the NameNode exactly like `create_operator`; `hdfs.name-node` entries are trimmed one by one and must carry a port; PyIceberg's `hdfs.host`/`hdfs.port` are accepted for authority-less paths. The rest is docs and tests (two-NameNode `delete_stream`, first-insert-wins cache, IPv6 parsing). Verified locally with the full fixture suite; description updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6345593584
Thanks everyone to work on this PR, it unblocks a vast number of use cases.
Teamwork 💪
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4187742576
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +261,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: HdfsNativeOperatorCache,
Review Comment:
This is @comphead's public-API point from round 1, still open: `operators` is a public field on a public variant, so `HdfsNativeOperatorCache` is now permanent semver surface and callers can construct the variant with an arbitrary cache — the siblings only expose `config: Arc<...>`. Folding both fields behind one opaque type (`HdfsNative(HdfsNativeStorage)` with private fields), or `#[non_exhaustive]` on the variant plus `pub(crate)` on the cache, keeps it off the stable surface. Worth settling before this is published rather than after.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
Review Comment:
`config.name_node.clone().or(authority_name_node)` means a configured `hdfs.name-node` wins over the authority in *every* path, so `hdfs://other-cluster:8020/...` silently routes to the configured cluster — a wrong-cluster read/write with no error. That's exactly right for an HA logical name, but it also swallows a real second cluster. I'd apply the configured name-node only when the path authority is absent or isn't a concrete `host:port` (i.e. a logical nameservice), and otherwise let the authority stand. If we'd rather keep the simple precedence, the `HDFS_NAME_NODE` doc needs to call out the all-paths-go-to-one-cluster behavior explicitly, with a test pinning the mixed-cluster case.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .or_else(|| hdfs_native_default_fs(config))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). The
+/// cache lives as long as the storage that owns it (clones share it) and
+/// never evicts, so the storage must not be used from another runtime once
+/// the building one is dropped: `hdfs-native` panics on a dead runtime.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
Review Comment:
The doc here is honest about the hazard but leaves it as a landmine. The cache never evicts, and each cached Operator pins its RPC tasks to whichever runtime was current when it was first built — so a FileIO/catalog that outlives that runtime (reused across `#[tokio::test]`s, rebuilt after a worker shuts down, shared across runtimes in a server) aborts the process the next time it touches HDFS, because hdfs-native panics on a dead runtime. That's reachable from a plain library call, which is the never-panic line I don't want us to cross.
Keying the cache on the runtime id and rebuilding on a miss, or just building per call like Gcs/Oss/Azdls, both close it. If it's genuinely out of scope for an experimental backend, I'd want this as a loud warning on the variant doc and the README rather than a comment on a private struct — plus a test that actually reproduces the cross-runtime abort.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
Review Comment:
We reject a configured `hdfs.name-node` without a port here, but a path authority (`hdfs://ns1/x`) and `hadoop.fs.defaultFS` (`hdfs://ns1`) both sail through portless, and `operator_build` then feeds that portless string straight to hdfs-native. So the two inputs disagree: either portless is a valid logical nameservice — in which case this check is too strict and rejects the natural HA spelling `hdfs://ns1` — or hdfs-native dials `host:port` and the portless authority fails at first I/O with a confusing error, the very thing this check exists to prevent. Worth confirming what hdfs-native actually does with a portless NameNode, then applying one rule to all three sources and pinning it with a test.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. Takes precedence over the path
+/// authority; when unset, the NameNode is derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// NameNode host for authority-less paths, as in PyIceberg; paths that carry
+/// an authority ignore it, as they do there. Combined with [`HDFS_PORT`] into
+/// Hadoop's `fs.defaultFS`. PyIceberg's `hdfs.user` and `hdfs.kerberos_ticket`
+/// have no equivalent: the client reads `HADOOP_USER_NAME` and the default
+/// Kerberos credential cache.
+pub const HDFS_HOST: &str = "hdfs.host";
+/// NameNode port for [`HDFS_HOST`]; defaults to `8020`.
Review Comment:
The doc is upfront that `hdfs.user` and `hdfs.kerberos_ticket` have no equivalent, which is good — but a config copied over from PyIceberg will then silently run as a different identity (`HADOOP_USER_NAME` / OS user / default ccache) with no signal. I'd lean toward a `log::warn!` when either key is present so the drop is at least visible, or an outright `FeatureUnsupported` if we'd rather fail loud. Do you see a clean path to mapping `hdfs.user` through hdfs-native's user hook, or is warn-and-drop the pragmatic line for now?
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .or_else(|| hdfs_native_default_fs(config))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). The
+/// cache lives as long as the storage that owns it (clones share it) and
+/// never evicts, so the storage must not be used from another runtime once
+/// the building one is dropped: `hdfs-native` panics on a dead runtime.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // The build reads the Hadoop XML config synchronously (~0.1 ms, once
+ // per NameNode), so it runs outside the lock and is not worth a
+ // blocking-thread hop. A racing first caller may build too; the loser
+ // is dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
Review Comment:
This is the synchronous Hadoop XML read I flagged last round — it's documented now, but still running on the executor thread. The `~0.1 ms` holds for a local `$HADOOP_CONF_DIR`; on a slow or networked mount it's a blocking call on an async worker, and when no `hdfs.name-node` is configured it fires once per distinct NameNode seen in metadata. I'd still rather wrap the build in `spawn_blocking`. If we're consciously accepting it for the experimental backend, that's a fine call — just say so at the call site rather than only describing the timing.
##########
.github/workflows/ci.yml:
##########
@@ -241,6 +241,9 @@ jobs:
- name: Start Docker containers
if: matrix.test-suite.name == 'default'
+ env:
+ # The HDFS fixture needs host networking; opt in here (Linux only).
+ COMPOSE_PROFILES: hdfs
Review Comment:
Opting the HDFS fixture in on the default suite means every default job now pulls the Hadoop image and waits on two container health checks (up to 30×5s) before any test runs — a heavy fixed cost on every PR for an experimental backend, and the whole workspace suite goes red if the fixture flakes. I'd gate this to a separate job (or to `crates/storage/opendal/**` changes) so an HDFS hiccup can't block unrelated work. Worth double-checking `make docker-up` passes `--wait` and honors the profile so a slow start fails cleanly rather than racing the tests.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
Review Comment:
Port handling is gated behind `hdfs.host` being set, so `hdfs.port` on its own is silently dropped and a bad value like `hdfs.port=x` is only caught when a host is also present. A `tracing::warn!` on a port with no host (or validating it regardless) would avoid a config that looks applied but isn't.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
Review Comment:
A bare `hadoop.` key strips to `""` and gets forwarded to opendal as an empty option key. Worth filtering out the empty stripped key here.
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,353 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the HDFS fixture in `dev/docker-compose.yaml` and are
+//! skipped when `ICEBERG_TEST_HDFS_ENDPOINT` is not set. The fixture uses
+//! host networking (Linux, or a Docker runtime that supports it), so it sits
+//! behind a compose profile:
+//!
+//! ```text
+//! COMPOSE_PROFILES=hdfs make docker-up
+//! ICEBERG_TEST_HDFS_ENDPOINT=hdfs://localhost:8020 cargo test -p iceberg-storage-opendal \
+//! --features opendal-hdfs-native --test file_io_hdfs_test
+//! ```
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::sync::Arc;
+
+ use bytes::Bytes;
+ use futures::StreamExt;
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::{OpenDalResolvingStorageFactory, OpenDalStorageFactory};
+ use iceberg_test_utils::{
+ ENV_HDFS_ENDPOINT, get_hdfs_endpoint, normalize_test_name_with_parts, set_up,
+ };
+
+ /// Skips the calling test unless the HDFS fixture endpoint is configured;
+ /// an unset *or* empty variable means "not provided" (see the HF tests).
+ macro_rules! require_hdfs {
+ () => {
+ match std::env::var(ENV_HDFS_ENDPOINT) {
+ Ok(v) if !v.is_empty() => {}
+ _ => {
+ eprintln!("Skipping HDFS test: {} not set", ENV_HDFS_ENDPOINT);
+ return;
+ }
+ }
+ };
+ }
+
+ fn get_file_io() -> FileIO {
+ set_up();
+ FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative)).build()
+ }
+
+ fn test_path(suffix: &str) -> String {
+ format!(
+ "{}/{}",
+ get_hdfs_endpoint(),
+ normalize_test_name_with_parts!(suffix)
+ )
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_exists() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let absent = test_path("test_file_io_hdfs_exists_absent");
+ assert!(!file_io.exists(&absent).await.unwrap());
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_write_and_read() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_write_and_read");
+ let _ = file_io.delete(&path).await;
+
+ let output = file_io.new_output(&path).unwrap();
+ output
+ .write(Bytes::from_static(b"hello hdfs"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ let input = file_io.new_input(&path).unwrap();
+ assert_eq!(
+ input.read().await.unwrap(),
+ Bytes::from_static(b"hello hdfs")
+ );
+ }
+
+ /// The HA flow: table locations carry a logical authority while
+ /// `hdfs.name-node` carries the (comma-separated) endpoints; it wins.
+ #[tokio::test]
+ async fn test_file_io_hdfs_configured_name_node() {
+ require_hdfs!();
+ set_up();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, get_hdfs_endpoint())
+ .build();
+
+ // The path authority is a logical name; the configured NameNode wins.
+ let path = format!(
+ "hdfs://logical-nameservice/{}",
+ normalize_test_name_with_parts!("test_file_io_hdfs_configured_name_node")
+ );
+ let _ = file_io.delete(&path).await;
+
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(b"via configured name node"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"via configured name node")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_overwrite() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_overwrite");
+ let _ = file_io.delete(&path).await;
+
+ for content in [b"first".as_slice(), b"second, longer".as_slice()] {
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(content))
+ .await
+ .unwrap();
+ }
+
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"second, longer")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let paths: Vec<String> = (0..5)
+ .map(|i| format!("{}/file-{i}", test_path("test_file_io_hdfs_delete_stream")))
+ .collect();
+ for path in &paths {
+ let _ = file_io.delete(path).await;
+ file_io
+ .new_output(path)
+ .unwrap()
+ .write("delete-me".into())
+ .await
+ .unwrap();
+ assert!(file_io.exists(path).await.unwrap());
+ }
+
+ let stream = futures::stream::iter(paths.clone()).boxed();
+ file_io.delete_stream(stream).await.unwrap();
+
+ for path in &paths {
+ assert!(!file_io.exists(path).await.unwrap());
+ }
+ }
+
+ /// Paths are batched per effective NameNode; two spellings of the fixture's
+ /// NameNode drive two batches through one stream. The fixture is one
+ /// cluster, so the keying itself is pinned by the unit tests.
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream_two_name_nodes() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let dir = test_path("test_file_io_hdfs_delete_stream_two_name_nodes");
+ let alt_dir = dir.replacen("localhost", "127.0.0.1", 1);
Review Comment:
`dir.replacen("localhost", "127.0.0.1", 1)` is a no-op unless the endpoint literally contains `localhost` — against `hdfs://namenode:8020` `alt_dir == dir`, both paths key to one batch, and the test passes having exercised nothing. An `assert_ne!(dir, alt_dir)` (or skipping when the endpoint doesn't contain `localhost`) would keep it honest. Worth confirming it's stable on the GH runners too — `localhost` vs `127.0.0.1` can resolve to different stacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4194185008
##########
.github/workflows/ci.yml:
##########
@@ -241,6 +241,9 @@ jobs:
- name: Start Docker containers
if: matrix.test-suite.name == 'default'
+ env:
+ # The HDFS fixture needs host networking; opt in here (Linux only).
+ COMPOSE_PROFILES: hdfs
Review Comment:
Measured on CI: the profile added ~40 s to "Start Docker containers" (108 s → 147 s) on every default run, and with `--wait` a fixture hiccup fails the suite before any test runs. Moved it to a dedicated `Tests (hdfs)` job that starts only the two HDFS containers and runs the HDFS test binary; the default job no longer pulls the image, and the HDFS tests self-skip there. I kept the new job in `ci-required` so the backend's tests stay gating, but a flake now costs a one-job re-run rather than the whole suite. One known limitation of the self-skip pattern (shared with the HF tests): the job only tests anything because it sets the endpoint variable, so that line is load-bearing. `--wait` and the profile were already in place. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6096350482
@blackmwk @comphead @laskoviymishka
Thank you for taking the time to review and provide feedback.
I believe I have addressed all the issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4346896261
##########
crates/storage/opendal/README.md:
##########
@@ -82,3 +85,40 @@ async fn main() -> iceberg::Result<()> {
Ok(())
}
```
+
+## HDFS
Review Comment:
Why we need to add this section for hdfs only? I'm not against it, but I think we should add docs for all storages. Please remove this section and create an issue as follow up to add them.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, LazyLock, PoisonError, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use iceberg_property_macro::Properties;
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Runtime;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| hdfs_native_endpoint(host, port))
+}
+
+/// An endpoint as every source spells it, the host lowercased because host
+/// names are case-insensitive: `NN1:8020` and `nn1:8020` share one client,
+/// as they share one `FileSystem` in Hadoop's cache.
+fn hdfs_native_endpoint(host: &str, port: u16) -> String {
+ format!("hdfs://{}:{port}", host.to_ascii_lowercase())
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Prefix of `hdfs.name-node.<nameservice>`, which declares one HA nameservice.
+const HDFS_NAME_NODE_DECLARATION_PREFIX: &str = "hdfs.name-node.";
+
+/// The properties this backend reads; unknown keys are ignored. Blank values
+/// count as unset, as in PyIceberg.
+#[derive(Properties)]
+struct HdfsNativeProperties {
+ /// `hdfs.name-node`, each entry normalized.
+ #[property(
+ key = HDFS_NAME_NODE,
+ default = Vec::new(),
+ parse_with = parse_name_node_list
+ )]
+ name_node: Vec<String>,
+ /// `hdfs.name-node.<nameservice>` declarations, keyed by nameservice.
+ #[property(prefix = HDFS_NAME_NODE_DECLARATION_PREFIX)]
+ nameservices: HashMap<String, String>,
+ #[property(key = HDFS_HOST, default = "", parse_with = parse_trimmed)]
+ host: String,
+ #[property(key = HDFS_PORT, default = "", parse_with = parse_trimmed)]
+ port: String,
+ /// `hadoop.*` keys with the prefix stripped, forwarded to the client.
+ #[property(prefix = HDFS_HADOOP_CONF_PREFIX)]
+ hadoop: HashMap<String, String>,
+}
+
+// Entries are trimmed one by one: opendal splits the list on `,` as is, so a
+// space after a comma would break failover to that NameNode. An empty list
+// stays unset because `Operator::from_config` bypasses the builder's
+// empty-string guard and `Some("")` would shadow the path-authority fallback.
+fn parse_name_node_list(value: &str) -> Result<Vec<String>> {
+ hdfs_native_name_node_list(HDFS_NAME_NODE, value)
+}
+
+fn parse_trimmed(value: &str) -> Result<String> {
+ Ok(value.trim().to_string())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let props = HdfsNativeProperties::from_properties(&m)?;
+ let mut options = props.hadoop;
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, value) in props.nameservices {
+ let key = format!("{HDFS_NAME_NODE_DECLARATION_PREFIX}{nameservice}");
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{HDFS_NAME_NODE_DECLARATION_PREFIX}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.contains_key(key) {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let port = match props.port.as_str() {
+ "" => None,
+ port => Some(port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}"),
+ )
+ .with_source(e)
+ })?),
+ };
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match props.host.as_str() {
+ "" if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ "" => {}
+ host => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.to_string()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ }
+
+ let mut cfg = HdfsNativeConfig::default();
+ if !props.name_node.is_empty() {
+ cfg.name_node = Some(props.name_node.join(","));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ // Every declaration, sugar or passed through, and the default filesystem
+ // fail here rather than at the first path that uses them.
+ for nameservice in cfg.options.iter().flat_map(|options| {
+ options.keys().filter_map(|key| {
+ key.strip_prefix(HA_NAMENODES_PREFIX)
+ .and_then(|rest| rest.strip_prefix('.'))
+ })
+ }) {
+ hdfs_native_nameservice(&cfg, nameservice)?;
+ }
+ hdfs_native_default_name_node(&cfg)?;
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ // Only a real scheme is kept; anything else before `://` may be userinfo.
+ let (scheme, rest) = match s.split_once("://") {
+ Some((scheme, rest))
+ if scheme
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c)) =>
+ {
+ (format!("{scheme}://"), rest)
+ }
+ _ => (String::new(), s),
+ };
+ match rest.rsplit_once('@') {
+ Some((_, after)) => format!("{scheme}***@{after}"),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// A path's authority, as the resolver reads it.
+#[derive(Debug, PartialEq)]
+pub(crate) enum HdfsNativeAuthority {
+ /// `hdfs://host:port`, normalized: dialed as is.
+ NameNode(String),
+ /// `hdfs://name` without a port: a nameservice that must be declared.
+ Nameservice(String),
+ /// `hdfs:///path`: served by `hdfs.name-node`, else `fs.defaultFS`.
+ Default,
+}
+
+/// Parse an HDFS path into its authority and the relative path (no leading
+/// `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(HdfsNativeAuthority, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let authority = match (url.host_str().filter(|h| !h.is_empty()), url.port()) {
+ (Some(host), Some(port)) => HdfsNativeAuthority::NameNode(hdfs_native_endpoint(host, port)),
+ (Some(host), None) => HdfsNativeAuthority::Nameservice(host.to_string()),
+ (None, _) => HdfsNativeAuthority::Default,
+ };
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Take the path from the original input instead: everything
+ // after the first `/` following the `hdfs://` prefix. Opendal paths must
+ // not start with `/` (`Deleter::delete` rejects them).
+ let rel = after_scheme
+ .split_once('/')
+ .map_or("", |(_, path)| path.trim_start_matches('/'));
+
+ Ok((authority, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = match authority {
+ HdfsNativeAuthority::NameNode(name_node) => name_node,
+ HdfsNativeAuthority::Nameservice(nameservice) => {
+ hdfs_native_nameservice(config, &nameservice)?.ok_or_else(|| {
+ hdfs_native_invalid_path(
+ path,
+ format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ),
+ )
+ })?
+ }
+ HdfsNativeAuthority::Default => hdfs_native_default_name_node(config)?.ok_or_else(|| {
+ hdfs_native_invalid_path(
+ path,
+ format!("authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"),
+ )
+ })?,
+ };
+ Ok((name_node, relative_path))
+}
+
+/// The NameNode that serves authority-less paths: `hdfs.name-node`, else
+/// `fs.defaultFS`, which must then be an HDFS `host:port`.
+fn hdfs_native_default_name_node(config: &HdfsNativeConfig) -> Result<Option<String>> {
+ if let Some(name_node) = &config.name_node {
+ return Ok(Some(name_node.clone()));
+ }
+ let Some(default_fs) = config
+ .options
+ .as_ref()
+ .and_then(|options| options.get(FS_DEFAULT_FS))
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+ else {
+ return Ok(None);
+ };
+ hdfs_native_name_node(default_fs).map(Some).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{FS_DEFAULT_FS}`: {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ),
+ )
+ })
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let mut name_nodes = Vec::new();
+ for id in ids.split(',').map(str::trim).filter(|id| !id.is_empty()) {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ let Some(name_node) = options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ ));
+ };
+ name_nodes.push(name_node);
+ }
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: one per endpoint or nameservice
+/// the storage's paths reach, which in practice is a handful of clusters, so
+/// it is not bounded. The cache lives as long as the storage that owns it
+/// (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+// A poisoned lock is recovered rather than reported: writes only insert
+// whole entries, so a panic elsewhere cannot leave the map broken.
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Option<Operator> {
+ self.0
+ .read()
+ .unwrap_or_else(PoisonError::into_inner)
+ .get(name_node)
+ .cloned()
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Operator {
+ self.0
+ .write()
+ .unwrap_or_else(PoisonError::into_inner)
+ .entry(name_node)
+ .or_insert(op)
+ .clone()
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+/// The runtime `hdfs-native` clients are built in. A client spawns its
+/// NameNode connections on the runtime current when it is built and panics
+/// once that runtime is gone, so building it here keeps a cached operator
+/// usable from any caller's runtime. Started on first use.
+// TODO: pass a runtime to the client instead of entering this one once
+// opendal's hdfs-native service exposes hdfs-native's
+// `ClientBuilder::with_io_runtime` (Kimahriman/hdfs-native#255).
+static HDFS_NATIVE_RUNTIME: LazyLock<Runtime> = LazyLock::new(|| {
Review Comment:
Please create an issue to pass catalog runtime to storage factor so that we could remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6347995691
Super team! Thanks @mixermt / @comphead / @blackmwk !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3893167714
##########
.github/workflows/ci.yml:
##########
@@ -232,6 +232,16 @@ jobs:
cargo test --no-fail-fast ${{ matrix.test-suite.args }}
fi
+ # HDFS tests are `#[ignore]`d (docker fixture needs Linux host
+ # networking); opt them in here by test-name filter.
+ - name: Run HDFS integration tests
Review Comment:
Done in 2b0ca95 — removed the step and the `#[ignore]`s; the suite now runs under the default `cargo nextest run` (the fixture is already started by `make docker-up`, and the Tests job is Linux-only).
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
Done in 2b0ca95. One note: with `pub(crate)` the struct has no non-test consumer inside the core crate yet (`iceberg-storage-opendal` shares only the key constants), so it carries `#[allow(dead_code)]` for now — happy to drop the struct entirely instead if you prefer.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
Done in 2b0ca95 — migrated to `#[derive(Properties)]` with `key`/`prefix` attributes and dropped the hand-written `TryFrom` + `TypedBuilder`/serde derives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3893188071
##########
crates/storage/opendal/src/hdfs.rs:
##########
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn parse_hdfs_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ if url.scheme() != "hdfs" {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let after_scheme = &path["hdfs://".len()..];
Review Comment:
Good catch — confirmed: `Url::parse("hdfs:x")` succeeds as a non-hierarchical URL and the slice panicked. Fixed in 45bd68a by requiring the literal `hdfs://` prefix before slicing, with regression tests for `hdfs:x`, `hdfs:/x`, and `hdfs:`.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -322,6 +353,10 @@ impl OpenDalStorage {
));
}
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ OpenDalStorage::Hdfs { config, operators } => {
Review Comment:
Confirmed and fixed in 45bd68a: `batch_key_for_path` now has an Hdfs arm keyed by the effective NameNode (configured `hdfs.name-node`, else the parsed authority including port), mirroring the operator-cache key, with unit tests covering the differing-port case and the configured-NameNode case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5811974531
Thanks @comphead. This is already an optional feature: `opendal-hdfs-native` on `iceberg-storage-opendal`. It is off by default (default is memory, fs, and s3) and not part of `opendal-all`. The backend is `cfg`-gated, and `services-hdfs-native` is only a dependency when the feature is on, so a normal Iceberg build does not compile it. A user opts in with:
```toml
iceberg-storage-opendal = { version = "...", features = ["opendal-hdfs-native"] }
```
If the goal is stronger than that — keeping `hdfs-native` out of this repo's `Cargo.lock`, or out of `--all-features` CI — a feature cannot do it, because the workspace lockfile and `--all-features` cover every feature. That would mean a separate crate this workspace does not depend on. I would rather not restructure unless that is what you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6101001942
@comphead Done: the `iceberg-storage-opendal` README now has an [HDFS section](https://github.com/mixermt/iceberg-rust/blob/0145ca6b6bce85a0cd52dda2636f0402e55969f8/crates/storage/opendal/README.md#hdfs) covering how to enable `opendal-hdfs-native`, the factory, the `hdfs.*`/`hadoop.*` properties, the Hadoop config, user and Kerberos it reads, and the portless-authority limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6034289512
Ah, there is a blocking change_request from @blackmwk , so i'll wait for him to stamp before merging at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6083718701
Thanks folks fore helping out, I'll do another round of review today as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4234496826
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
Review Comment:
Done: `hdfs.host`/`hdfs.port` now go through the `Properties` struct and a `match`, the `??` went away with `spawn_blocking`, and the nameservice lookup is a `for` loop with `let ... else`.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
Review Comment:
Done: the prefix slicing went away with the `Properties` struct, and `hdfs_native_redact` and the path parser now use `split_once`/`rsplit_once`, so no index slicing is left.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
Review Comment:
Done: the port parse error is attached with `with_source` now; it was the only one interpolated.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
Review Comment:
Confirmed both. It is expected to stay small: one entry per endpoint or nameservice the storage's paths reach, so a handful of clusters, unbounded like Hadoop's own `FileSystem` cache; the doc says so now, and with the sentinel gone an entry is just the operator and its client. The case split was real (`host_str` keeps case for `hdfs://`), so endpoint hosts are lowercased now, as Hadoop's cache key does, and `hdfs://NN1:8020` and `hdfs://nn1:8020` share one client. Nameservice names stay case-sensitive, like the Hadoop keys that declare them.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
Review Comment:
The sentinel is gone (see the `spawn_blocking` thread): clients are built inside a crate-owned runtime that is never dropped, so there is no liveness check left to race, and a test reproduces the old panic when they are built on the caller's runtime instead. I added a TODO there: hdfs-native already takes an external runtime (Kimahriman/hdfs-native#255), but opendal's hdfs-native service does not expose it yet; once it does, the runtime can be passed in instead of entered.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -186,6 +194,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
Agreed it belongs in its own PR: `#[non_exhaustive]` is itself a breaking change for the same exhaustive matches, and it covers every backend's variant, not just this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4234494910
##########
crates/iceberg/src/io/storage/config/hdfs_native.rs:
##########
Review Comment:
Done: the module is now `hdfs_native`, matching its siblings (`azdls`, `gcs`, `hf`, `oss`, `s3`). It is private and re-exported, so the public `iceberg::io::HDFS_*` paths are unchanged.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -1159,6 +1201,102 @@ mod tests {
);
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ fn hdfs_native_test_storage() -> OpenDalStorage {
+ OpenDalStorage::HdfsNative(HdfsNativeStorage::new(
+ opendal::services::HdfsNativeConfig::default(),
+ OpenDalClientConfig::default(),
+ ))
+ }
+
+ /// The configuration round-trips through serde; the operator cache does
+ /// not and starts empty.
+ #[cfg(feature = "opendal-hdfs-native")]
+ #[tokio::test]
+ async fn test_hdfs_native_storage_serde_round_trip() {
Review Comment:
Done: both tests moved to `hdfs_native` with the module's `test_hdfs_native_` prefix. Opened #3385 for splitting the remaining `lib.rs` tests.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
Review Comment:
Done: the parser now reads `hdfs.name-node`, the `hdfs.name-node.<nameservice>` and `hadoop.` prefixes, `hdfs.host` and `hdfs.port` through `#[derive(Properties)]`. The validation on top is unchanged.
##########
.github/workflows/ci.yml:
##########
@@ -265,6 +265,46 @@ jobs:
if: always() && matrix.test-suite.name == 'default'
run: make docker-down
+ tests-hdfs:
Review Comment:
Done: the HDFS fixture now follows the other fixtures. It runs on the shared bridge network with published ports, so `make docker-up` starts it on every platform (the DataNode advertises `127.0.0.1` and the tests set `dfs.client.use.datanode.hostname`), the tests always run, and the dedicated job is gone. Trade-off: the default job pays the Hadoop startup again (~40 s), which @laskoviymishka had asked to move out of it.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator, handle: &Handle) -> Self {
+ Self {
+ operator,
+ sentinel: RuntimeSentinel::spawn(handle),
+ }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel.alive.strong_count() > 0
+ }
+}
+
+// A poisoned lock is recovered rather than reported: writes only ever
+// replace whole entries, so a panic elsewhere cannot leave the map broken.
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Option<Operator> {
+ self.0
+ .read()
+ .unwrap_or_else(PoisonError::into_inner)
+ .get(name_node)
+ .filter(|cached| cached.runtime_alive())
+ .map(|cached| cached.operator.clone())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds; a stale entry is replaced.
+ fn insert(&self, name_node: String, op: Operator, handle: &Handle) -> Operator {
+ let mut operators = self.0.write().unwrap_or_else(PoisonError::into_inner);
+ match operators.entry(name_node) {
+ Entry::Occupied(entry) if entry.get().runtime_alive() => entry.get().operator.clone(),
+ Entry::Occupied(mut entry) => {
+ entry.insert(CachedOperator::new(op.clone(), handle));
+ op
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(CachedOperator::new(op.clone(), handle));
+ op
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) async fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &Arc<HdfsNativeConfig>,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node) {
+ return Ok((op, relative_path));
+ }
+
+ // Every operator in this crate needs a tokio runtime for its I/O (the
+ // timeout layer), so say so instead of panicking in `spawn_blocking`.
+ let handle = Handle::try_current().map_err(|_| {
+ Error::new(
+ ErrorKind::FeatureUnsupported,
+ "HDFS storage requires a tokio runtime",
+ )
+ })?;
+
+ // The build reads the Hadoop XML config synchronously, so it runs on a
+ // blocking thread and outside the lock. A racing first caller may build
+ // too; the loser is dropped before opening any connection.
+ let build_config = Arc::clone(config);
+ let build_name_node = name_node.clone();
+ let op = handle
Review Comment:
Done, and the runtime handle is gone too. An hdfs-native client spawns its NameNode connections on the runtime current when it is built and panics once that runtime is dropped, so clients are now built inside a crate-owned runtime and work from any caller's runtime: no `Handle::try_current`, sentinel or `spawn_blocking` remain, and `create_operator` is sync again. The Hadoop XML read (~0.1 ms) runs inline, as @laskoviymishka allowed for, with a note at the call site. `FileIO` has no runtime to pass today; once it carries `iceberg::runtime::Runtime`, the build can use its io handle instead.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -372,7 +394,7 @@ impl OpenDalStorage {
/// * An [`opendal::Operator`] instance used to operate on file.
/// * Relative path to the root uri of [`opendal::Operator`].
#[allow(unreachable_code, unused_variables)]
- pub(crate) fn create_operator<'a>(
+ pub(crate) async fn create_operator<'a>(
Review Comment:
Done: `create_operator` is sync again for every backend, as on main; the HDFS build no longer needs a runtime (see the reply to blackmwk above).
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
Review Comment:
Done in part. Paths now parse once into an authority enum (`NameNode`, `Nameservice`, `Default`), so the resolver no longer re-parses the authority, and every declaration, including passed-through `hadoop.dfs.ha.namenodes.*` and the `fs.defaultFS` that serves authority-less paths, is validated when the storage is built. I kept the declarations in opendal's options rather than a second, derived copy in the serialized storage, since a lookup is a few map reads next to an RPC, and kept one file per backend like the other backends.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
Review Comment:
The authority is an enum now (see above). I kept the `hdfs_native_` free functions rather than methods, matching the other backends and @blackmwk's earlier request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4194234355
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. Takes precedence over the path
+/// authority; when unset, the NameNode is derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// NameNode host for authority-less paths, as in PyIceberg; paths that carry
+/// an authority ignore it, as they do there. Combined with [`HDFS_PORT`] into
+/// Hadoop's `fs.defaultFS`. PyIceberg's `hdfs.user` and `hdfs.kerberos_ticket`
+/// have no equivalent: the client reads `HADOOP_USER_NAME` and the default
+/// Kerberos credential cache.
+pub const HDFS_HOST: &str = "hdfs.host";
+/// NameNode port for [`HDFS_HOST`]; defaults to `8020`.
Review Comment:
Warn-and-drop it is: opendal's `HdfsNativeConfig` has no user field and hdfs-native has no config key for it (`with_user` exists on its builder, but opendal never calls it), so mapping would need an opendal change first. An error would reject every PyIceberg config that just carries `hdfs.user`. Both keys now log a warning naming the key, and a test pins that they are never forwarded as Hadoop options. Lands with the next push.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
Review Comment:
`hdfs.port` is now validated whenever it is present, host or not, and a valid port without `hdfs.host` logs a warning instead of vanishing. Tests cover both. Lands with the next push.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
Review Comment:
Fixed: a bare `hadoop.` key is rejected with a `DataInvalid` naming the prefix, with a test. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5780605775
It would be nice to see this PR merged, as the Iceberg+HDFS still in demand.
@blackmwk @findepi @huaxingao @Xuanwo @kevinjqliu appreciate if you can help or share your vision how community feel about this direction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4019032222
##########
dev/docker-compose.yaml:
##########
@@ -147,6 +147,49 @@ services:
timeout: 5s
retries: 5
+ # =============================================================================
+ # HDFS - single-node NameNode + DataNode for HDFS tests
+ # =============================================================================
+ # hdfs-native connects to DataNodes by their registered IP — unroutable on
+ # a docker bridge, hence host networking (needs Linux or a recent runtime).
+ hdfs-namenode:
+ image: apache/hadoop:3.5.0
+ network_mode: "host"
+ command: ["hdfs", "namenode"]
+ environment:
+ ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
+ extra_hosts:
+ - "docker-desktop:127.0.0.1"
+ volumes:
+ - ./hdfs/core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml:ro
+ - ./hdfs/hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml:ro
+ healthcheck:
+ test: ["CMD-SHELL", "hdfs dfsadmin -safemode get | grep -q OFF"]
+ interval: 5s
+ timeout: 15s
+ retries: 30
+ start_period: 30s
+
+ hdfs-datanode:
Review Comment:
interesting can we test on multi datanodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3893105048
##########
crates/storage/opendal/src/hdfs.rs:
##########
@@ -0,0 +1,289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn parse_hdfs_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ if url.scheme() != "hdfs" {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let after_scheme = &path["hdfs://".len()..];
Review Comment:
`Url::parse("hdfs:x")` succeeds because non-special schemes may be non-hierarchical, but this unconditional byte-7 slice then panics (`start byte index 7 is out of bounds`) instead of returning `DataInvalid`. Please reject non-hierarchical URLs or require the `hdfs://` prefix before slicing, and add a regression test for this case.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -322,6 +353,10 @@ impl OpenDalStorage {
));
}
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ OpenDalStorage::Hdfs { config, operators } => {
Review Comment:
The HDFS operator cache distinguishes the effective NameNode including its port, but `delete_stream` still groups paths through `batch_key_for_path`, which uses only `host_str()`. Consequently, `hdfs://namenode:8020/a` and `hdfs://namenode:9000/b` share one deleter and one of the deletes is sent through the wrong operator. Please special-case the HDFS batch key to use the configured NameNode or the parsed authority including port, with regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4194082722
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +261,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: HdfsNativeOperatorCache,
Review Comment:
Done: the variant is now `HdfsNative(HdfsNativeStorage)`, one public struct with crate-private fields and no public constructor, so only the factories build it and the cache type is no longer public (`HdfsNativeOperatorCache` is `pub(crate)`). The `#[non_exhaustive]` plus `pub(crate)` field route does not compile (E0446, the field type would be more private than the variant). The serialized form keeps the same nesting, which the new round-trip test pins, and `HdfsNative` now takes part in `test_factory_propagates_io_timeout`, which had listed every factory but this one. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4197130885
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
+ name_node
+ .rsplit_once(':')
+ .is_some_and(|(_, port)| port.parse::<u16>().is_ok())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // Each entry is dialed as `host:port` (`hdfs://` optional); a portless
+ // one would fail only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| !hdfs_native_has_port(entry)) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_has_port(&authority) => authority,
+ Some(logical) => config.name_node.clone().ok_or_else(|| {
+ let logical = logical.trim_start_matches("hdfs://");
+ invalid(format!(
+ "logical nameservice `{logical}` requires `{HDFS_NAME_NODE}`"
+ ))
+ })?,
+ None => match config
+ .name_node
+ .clone()
+ .or_else(|| hdfs_native_default_fs(config))
+ {
+ Some(name_node) if hdfs_native_has_port(&name_node) => name_node,
+ Some(logical) => {
+ return Err(invalid(format!(
+ "`{FS_DEFAULT_FS}` {logical} has no port, a logical nameservice requires `{HDFS_NAME_NODE}`"
+ )));
+ }
+ None => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). An
+/// entry is rebuilt once that runtime is gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ /// `None` when built outside any runtime.
+ sentinel: Option<RuntimeSentinel>,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator) -> Self {
+ let sentinel = Handle::try_current()
+ .ok()
+ .map(|handle| RuntimeSentinel::spawn(&handle));
+ Self { operator, sentinel }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel
+ .as_ref()
+ .is_none_or(|sentinel| sentinel.alive.strong_count() > 0)
+ }
+}
+
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self
+ .0
+ .read()
+ .map_err(poisoned)?
+ .get(name_node)
+ .filter(|cached| cached.runtime_alive())
+ .map(|cached| cached.operator.clone()))
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds; a stale entry is replaced.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ let mut operators = self.0.write().map_err(poisoned)?;
+ match operators.entry(name_node) {
+ Entry::Occupied(entry) if entry.get().runtime_alive() => {
+ Ok(entry.get().operator.clone())
+ }
+ Entry::Occupied(mut entry) => {
+ entry.insert(CachedOperator::new(op.clone()));
+ Ok(op)
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(CachedOperator::new(op.clone()));
+ Ok(op)
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) async fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &Arc<HdfsNativeConfig>,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // The build reads the Hadoop XML config synchronously, so it runs on a
+ // blocking thread and outside the lock. A racing first caller may build
+ // too; the loser is dropped before opening any connection.
+ let build_config = Arc::clone(config);
+ let build_name_node = name_node.clone();
+ let op = tokio::task::spawn_blocking(move || {
Review Comment:
The crate isn't executor-agnostic today: every backend's I/O goes through opendal's `TimeoutLayer`, which uses `tokio::time`, so the memory backend panics the same way under `futures::executor::block_on` (checked). Building inline without a runtime would therefore only move the panic to the first I/O. What I did instead: `create_operator` now checks `Handle::try_current()` and returns `FeatureUnsupported` ("HDFS storage requires a tokio runtime") rather than panicking, with a test driving it under the futures executor. The dead branch is gone: the handle is passed into the cache, the sentinel is no longer optional, and the comment went with it.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
+ name_node
+ .rsplit_once(':')
+ .is_some_and(|(_, port)| port.parse::<u16>().is_ok())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // Each entry is dialed as `host:port` (`hdfs://` optional); a portless
+ // one would fail only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| !hdfs_native_has_port(entry)) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_has_port(&authority) => authority,
+ Some(logical) => config.name_node.clone().ok_or_else(|| {
Review Comment:
Confirmed from opendal's source (0.58.1 and 0.58.2): it always builds the client with the URL `hdfs://nameservice` and synthesizes that nameservice from the comma list, so a bare logical `name_node` is dialed as a socket address and never resolved from the XML. hdfs-native could resolve it, so that is an opendal limitation worth an upstream issue. So I went with your second branch: `hdfs.name-node.<nameservice>` declares a nameservice's own list; it expands to Hadoop's `dfs.ha.namenodes` / `dfs.namenode.rpc-address` keys in the forwarded options, which the resolver reads back, so declaring a nameservice directly through `hadoop.*` works too. Plain `hdfs.name-node` remains the single-cluster default for undeclared names and authority-less paths, documented loudly on the constant. Verified against two local listeners that each declared nameservice reaches its own NameNode and a concrete authority is never redirected; an undeclared name without the plain key errors before any I/O.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
Review Comment:
All five reproduced, including `::1` parsing as host `:` port 1. Replaced the helper with a normalizer that parses each entry as a URL authority after stripping an optional `hdfs://`, rejects other schemes, userinfo, paths, port 0 and unbracketed IPv6, and normalizes to `hdfs://host:port`, the form path authorities use, so `nn:8020` and `hdfs://nn:8020` now share one cache key (test added). The same function classifies path authorities and `fs.defaultFS`, so one rule covers all three sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4194370830
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,353 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the HDFS fixture in `dev/docker-compose.yaml` and are
+//! skipped when `ICEBERG_TEST_HDFS_ENDPOINT` is not set. The fixture uses
+//! host networking (Linux, or a Docker runtime that supports it), so it sits
+//! behind a compose profile:
+//!
+//! ```text
+//! COMPOSE_PROFILES=hdfs make docker-up
+//! ICEBERG_TEST_HDFS_ENDPOINT=hdfs://localhost:8020 cargo test -p iceberg-storage-opendal \
+//! --features opendal-hdfs-native --test file_io_hdfs_test
+//! ```
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::sync::Arc;
+
+ use bytes::Bytes;
+ use futures::StreamExt;
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::{OpenDalResolvingStorageFactory, OpenDalStorageFactory};
+ use iceberg_test_utils::{
+ ENV_HDFS_ENDPOINT, get_hdfs_endpoint, normalize_test_name_with_parts, set_up,
+ };
+
+ /// Skips the calling test unless the HDFS fixture endpoint is configured;
+ /// an unset *or* empty variable means "not provided" (see the HF tests).
+ macro_rules! require_hdfs {
+ () => {
+ match std::env::var(ENV_HDFS_ENDPOINT) {
+ Ok(v) if !v.is_empty() => {}
+ _ => {
+ eprintln!("Skipping HDFS test: {} not set", ENV_HDFS_ENDPOINT);
+ return;
+ }
+ }
+ };
+ }
+
+ fn get_file_io() -> FileIO {
+ set_up();
+ FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative)).build()
+ }
+
+ fn test_path(suffix: &str) -> String {
+ format!(
+ "{}/{}",
+ get_hdfs_endpoint(),
+ normalize_test_name_with_parts!(suffix)
+ )
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_exists() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let absent = test_path("test_file_io_hdfs_exists_absent");
+ assert!(!file_io.exists(&absent).await.unwrap());
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_write_and_read() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_write_and_read");
+ let _ = file_io.delete(&path).await;
+
+ let output = file_io.new_output(&path).unwrap();
+ output
+ .write(Bytes::from_static(b"hello hdfs"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ let input = file_io.new_input(&path).unwrap();
+ assert_eq!(
+ input.read().await.unwrap(),
+ Bytes::from_static(b"hello hdfs")
+ );
+ }
+
+ /// The HA flow: table locations carry a logical authority while
+ /// `hdfs.name-node` carries the (comma-separated) endpoints; it wins.
+ #[tokio::test]
+ async fn test_file_io_hdfs_configured_name_node() {
+ require_hdfs!();
+ set_up();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, get_hdfs_endpoint())
+ .build();
+
+ // The path authority is a logical name; the configured NameNode wins.
+ let path = format!(
+ "hdfs://logical-nameservice/{}",
+ normalize_test_name_with_parts!("test_file_io_hdfs_configured_name_node")
+ );
+ let _ = file_io.delete(&path).await;
+
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(b"via configured name node"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"via configured name node")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_overwrite() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_overwrite");
+ let _ = file_io.delete(&path).await;
+
+ for content in [b"first".as_slice(), b"second, longer".as_slice()] {
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(content))
+ .await
+ .unwrap();
+ }
+
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"second, longer")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let paths: Vec<String> = (0..5)
+ .map(|i| format!("{}/file-{i}", test_path("test_file_io_hdfs_delete_stream")))
+ .collect();
+ for path in &paths {
+ let _ = file_io.delete(path).await;
+ file_io
+ .new_output(path)
+ .unwrap()
+ .write("delete-me".into())
+ .await
+ .unwrap();
+ assert!(file_io.exists(path).await.unwrap());
+ }
+
+ let stream = futures::stream::iter(paths.clone()).boxed();
+ file_io.delete_stream(stream).await.unwrap();
+
+ for path in &paths {
+ assert!(!file_io.exists(path).await.unwrap());
+ }
+ }
+
+ /// Paths are batched per effective NameNode; two spellings of the fixture's
+ /// NameNode drive two batches through one stream. The fixture is one
+ /// cluster, so the keying itself is pinned by the unit tests.
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream_two_name_nodes() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let dir = test_path("test_file_io_hdfs_delete_stream_two_name_nodes");
+ let alt_dir = dir.replacen("localhost", "127.0.0.1", 1);
Review Comment:
Right, that was a no-op away from `localhost`. The second spelling is now the endpoint host resolved to its IPv4 address, so any hostname endpoint yields a second batch key; the test asserts the two spellings differ and skips explicitly when the endpoint is already an IP literal. Verified both paths against the fixture, and the earlier CI run already exercised `localhost` vs `127.0.0.1` green on the GitHub runners. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5554527527
@blackmwk
I've addressed all the recent feedback.
Could you please give this one last review? 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4018997879
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
+ Some(name_node) => name_node,
+ None => {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ ));
+ }
+ };
+
+ // Fast path: check read lock first.
+ {
+ let cache = operators
+ .read()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ if let Some(op) = cache.get(&name_node) {
+ return Ok((op.clone(), relative_path));
+ }
+ }
+
+ // Slow path: build and insert under write lock, re-checking for a
+ // concurrent insert.
+ let mut cache = operators
+ .write()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ let op = match cache.get(&name_node) {
+ Some(op) => op.clone(),
+ None => {
+ let op = hdfs_native_operator_build(config, &name_node)?;
Review Comment:
**Blocker: blocking I/O runs while the cache write lock is held.**
`hdfs_native_operator_build` -> `Operator::from_config` -> `HdfsNativeBuilder::build()` -> `hdfs_native::ClientBuilder::build()` -> `Configuration::new(..)`, which does synchronous `fs::read_to_string` on `core-site.xml`/`hdfs-site.xml` (hdfs-native 0.14.5, `src/common/config.rs:228`), followed by `NameServiceProxy::new`. All of that happens on a tokio worker thread, under `operators.write()`.
Two effects: the worker thread is blocked on file I/O, and every concurrent caller is serialized behind this lock, including callers that would have hit the cache for a *different*, already-built NameNode.
The double-checked-locking shape mirrors `OpenDalResolvingStorage::resolve`, but there the build is pure CPU. The cost profile is different here.
Suggested: build outside the lock and insert with `entry(..).or_insert_with(..)`, accepting the occasional duplicate build, or move the build into `spawn_blocking`.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
Review Comment:
**Major: an empty `hdfs.name-node` silently defeats the path-authority fallback.**
`hdfs_native_config_parse` (L35-37) accepts `""` as-is, so `config.name_node` becomes `Some("")` and `.or(authority_name_node)` short-circuits, meaning the authority is never consulted. `init_hdfs_config("")` then filters out the empty entry and emits `dfs.ha.namenodes.nameservice = ""`, producing a confusing failure well downstream instead of the pointed error this function is otherwise careful to give.
Worth noting that opendal's own `HdfsNativeBuilder::name_node()` guards this with `if !name_node.is_empty()`, but `Operator::from_config` sets the config struct directly and bypasses the setter, so that guard does not apply here.
Suggested: in `hdfs_native_config_parse`, `.filter(|s| !s.trim().is_empty())` before assigning. Trimming a trailing `/` there as well would stop `hdfs://nn:8020/` and a `hdfs://nn:8020` path authority from producing two cache entries for the same cluster.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
+ Some(name_node) => name_node,
+ None => {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ ));
+ }
+ };
+
+ // Fast path: check read lock first.
+ {
+ let cache = operators
+ .read()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ if let Some(op) = cache.get(&name_node) {
+ return Ok((op.clone(), relative_path));
+ }
+ }
+
+ // Slow path: build and insert under write lock, re-checking for a
+ // concurrent insert.
+ let mut cache = operators
+ .write()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ let op = match cache.get(&name_node) {
+ Some(op) => op.clone(),
+ None => {
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ cache.insert(name_node, op.clone());
+ op
+ }
+ };
+
+ Ok((op, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, mirroring the operator-cache key so paths that resolve to
+/// different operators never share a deleter.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
Review Comment:
**Major: the NameNode-precedence rule is implemented twice.**
`hdfs_native_batch_key` here and `hdfs_native_create_operator` (L99) each independently compute "configured `name_node`, else path authority". These two have to agree: `delete_stream` uses `batch_key_for_path` to pick the deleter and `create_operator` to build it, so if the rules ever drift, paths get grouped onto a deleter built against a different NameNode and deletes silently go to the wrong cluster.
Suggested: extract a single `fn hdfs_native_effective_name_node(config: &HdfsNativeConfig, path: &str) -> Result<String>` and have both call it. That also makes `test_hdfs_native_batch_key_configured_name_node_wins` and `test_hdfs_native_create_operator_configured_name_node_wins` redundant, since they currently assert the same rule twice.
Separately, `unwrap_or_default()` maps both "unparsable path" and "authority-less with no property" to `""`, so those share a grouping key. Harmless today because `create_operator` then errors on the same path, but it is a silent coupling worth not relying on.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +262,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: Arc<RwLock<HashMap<String, Operator>>>,
Review Comment:
**Major: the operator cache becomes part of the public API.**
`public-api.txt` gains:
```
pub iceberg_storage_opendal::OpenDalStorage::HdfsNative::operators: alloc::sync::Arc<std::sync::poison::rwlock::RwLock<std::collections::hash::map::HashMap<alloc::string::String, opendal_core::types::operator::operator::Operator>>>
```
No other variant exposes internal state; they all carry only `config`. This pins the cache representation (including the choice of `std::sync::RwLock`) as a semver-relevant detail, so swapping in a different cache later becomes a breaking change.
Suggested: wrap it in a newtype with private fields, e.g. `pub struct OperatorCache(RwLock<HashMap<String, Operator>>)`, so only an opaque type appears in the public API.
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the `hdfs-namenode`/`hdfs-datanode` services from
+//! `dev/docker-compose.yaml` (started by `make docker-up`); the fixture
+//! uses host networking, which needs Linux or a recent Docker runtime.
+
+#[cfg(feature = "opendal-hdfs-native")]
Review Comment:
**Blocker: these tests are not `#[ignore]`d, and there is no CI opt-in.**
The PR description says "Tests are `#[ignore]`d (host networking is Linux-only); CI opts in via `cargo nextest --run-ignored=only -E 'test(file_io_hdfs)'`". Neither is on the branch:
- no `#[ignore]` on any of the 12 tests in this file;
- `git diff main...HEAD -- .github/ Makefile` is empty, so there is no workflow step and no `--run-ignored=only`;
- `dev/docker-compose.yaml` adds `hdfs-namenode`/`hdfs-datanode` with no `profiles:` key, so `make docker-up` starts them unconditionally.
Since the `Makefile` has `test: docker-up` and `ci.yml` runs `--all-targets --all-features`, `opendal-hdfs-native` is enabled and all 12 tests run by default. On macOS/Windows, `network_mode: "host"` does not put the NameNode on the host loopback, so `make test` will fail for every contributor on those platforms.
Either add `#[ignore]` plus a Linux-only CI step as described, or put the two compose services behind a `profiles:` key so they are opt-in.
While here, the description's test counts look stale too: 16 unit tests in `hdfs_native.rs`, 3 in `lib.rs`, 2 in `resolving.rs` (not 24), and `crates/iceberg/src/io/storage/config/hdfs.rs` has no tests (not 3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4204790164
##########
crates/storage/opendal/tests/hdfs_native_runtime_test.rs:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A cached HDFS operator is bound to the tokio runtime that built it; once
+//! that runtime is gone it must be rebuilt rather than reused. Needs no
+//! HDFS: the NameNode is a local listener that only counts dials.
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::net::TcpListener;
+ use std::sync::Arc;
+ use std::sync::atomic::{AtomicUsize, Ordering};
+ use std::time::Duration;
+
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::OpenDalStorageFactory;
+
+ /// Accepts and immediately closes connections, counting them.
+ fn fake_name_node() -> (u16, Arc<AtomicUsize>) {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let port = listener.local_addr().unwrap().port();
+ let dials = Arc::new(AtomicUsize::new(0));
+ let counter = dials.clone();
+ std::thread::spawn(move || {
+ for stream in listener.incoming() {
+ let Ok(_stream) = stream else { break };
+ counter.fetch_add(1, Ordering::SeqCst);
+ }
+ });
+ (port, dials)
+ }
+
+ fn stat(runtime: &tokio::runtime::Runtime, file_io: &FileIO) {
+ runtime.block_on(async {
+ let input = file_io.new_input("hdfs:///f").unwrap();
+ // The fake NameNode never answers, so this fails; only the dial matters.
+ let _ = tokio::time::timeout(Duration::from_secs(10), input.metadata()).await;
+ });
+ }
+
+ #[test]
+ fn test_hdfs_operator_is_rebuilt_after_its_runtime_is_dropped() {
+ let (port, dials) = fake_name_node();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, format!("hdfs://127.0.0.1:{port}"))
+ .with_prop("hadoop.dfs.client.failover.max.attempts", "1")
+ .build();
+
+ let first = tokio::runtime::Runtime::new().unwrap();
+ stat(&first, &file_io);
+ let after_first = dials.load(Ordering::SeqCst);
+ assert!(after_first >= 1, "the NameNode was never dialed");
+ drop(first);
+
+ // Reusing the FileIO from another runtime used to panic inside
+ // hdfs-native, whose client was bound to the dropped runtime.
+ let second = tokio::runtime::Runtime::new().unwrap();
+ stat(&second, &file_io);
+ assert!(
+ dials.load(Ordering::SeqCst) > after_first,
Review Comment:
Nit — this can pass without a dial from the second runtime. `after_first` is sampled right after the 10s timeout, and the counter is bumped asynchronously by the listener thread, so a straggling retry from the first runtime satisfies `> after_first` on its own. I'd let the counter settle before sampling `after_first`, and ideally assert the cache entry's identity actually changed (a test-only generation counter, or the `Operator` pointer) so we're proving the rebuild rather than just a dial.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ // Not echoing the path: it may carry a password.
+ let host = url.host_str().unwrap_or_default();
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path for host `{host}`: userinfo is not supported"),
+ ));
+ }
+ if url.port() == Some(0) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, port 0 cannot be dialed"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through its declaration, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
+ let nameservice = logical.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "logical nameservice `{nameservice}` is not declared; set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {default_fs} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`"
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator, handle: &Handle) -> Self {
+ Self {
+ operator,
+ sentinel: RuntimeSentinel::spawn(handle),
+ }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel.alive.strong_count() > 0
+ }
+}
+
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self
+ .0
+ .read()
+ .map_err(poisoned)?
Review Comment:
Nit, and separate from the lock thread from last round (that one's fine — no guard held across an `.await`): a poisoned lock here turns every subsequent HDFS call into a permanent `Unexpected` error. The map is only ever mutated as whole-entry operations, so a poison just means some other thread panicked — it shouldn't take down all HDFS I/O with it. I'd recover the guard with `unwrap_or_else(PoisonError::into_inner)` at both the read and write sites instead of mapping it to an error.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
Review Comment:
Nit — the comment a few lines down says we don't echo the path because it may carry a password, but this `Url::parse` failure (and the scheme-mismatch `return` right below it) both format the full `{path}`. `hdfs://user:pw@nn:bad/x` fails `parse` before the userinfo check runs, so the password lands in the error. I'd redact userinfo before echoing, or echo only the scheme plus the parser reason.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ // Not echoing the path: it may carry a password.
+ let host = url.host_str().unwrap_or_default();
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path for host `{host}`: userinfo is not supported"),
+ ));
+ }
+ if url.port() == Some(0) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, port 0 cannot be dialed"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through its declaration, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
Review Comment:
Followup, not blocking — this arm treats any portless authority as a logical nameservice that must be declared through a property, and errors otherwise. The common HA deployment writes Iceberg locations as `hdfs://nameservice1/warehouse/...`, where `nameservice1` is defined only in the cluster's hdfs-site.xml under `HADOOP_CONF_DIR` — those tables fail here unless the user re-declares every nameservice as `hdfs.name-node.<ns>`.
hdfs-native already reads `HADOOP_CONF_DIR` and resolves nameservices itself, so I'd rather not pre-empt it with an error: passing `hdfs://<authority>` straight through as the `name_node` and letting the client resolve it (failing with its own error if it can't) would cover this, and `hdfs://host/path` with no port, which Hadoop reads as `host:8020`.
Fine to take as a follow-on. If the strict behavior is deliberate for now, the "as in Hadoop" line in the `HDFS_NAME_NODE` doc should go — it diverges here — and the limitation is worth a README note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5776162564
@comphead @blackmwk any other comments?
I really want to push this forward
I'm testing this with [datafusion-comet](https://github.com/apache/datafusion-comet/pull/5898), so far no issues in this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4174483348
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
Review Comment:
Good point, and the crate's convention is to use PyIceberg's keys, so I added `hdfs.host`/`hdfs.port` (port defaults to 8020 like pyarrow). I kept PyIceberg's semantics for them: they apply only to authority-less paths, since PyIceberg itself uses the URI's netloc first and only reads these props when there is none. Internally they map to Hadoop's `fs.defaultFS`, so `hadoop.fs.defaultFS` works the same way. `hdfs.name-node` remains the explicit override, which HA logical names need with hdfs-native. `hdfs.user`/`hdfs.kerberos_ticket` have no equivalent in opendal's config; hdfs-native reads `HADOOP_USER_NAME` and the default Kerberos cache, which the `HDFS_HOST` doc now states. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5878510220
Would love to see this merged too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6019477113
Thanks @laskoviymishka. The three threads from this round are addressed in 9c62f4c..ce78ee3, each with a reply inline: a missing tokio runtime is now a `FeatureUnsupported` error instead of a panic (and the unreachable branch is gone); every NameNode spelling is normalized to `hdfs://host:port` with the malformed shapes rejected; and nameservices can be declared with `hdfs.name-node.<nameservice>` (Hadoop's own keys underneath), with plain `hdfs.name-node` as the documented single-cluster default. Description updated; full checks and the fixture suite green locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6130662050
@blackmwk PTAL as you requested changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5805270566
One thing comes to my mind is we prob can rework the PR and make `hdfs` as an optional Rust feature, which IMO would make much more sense, so the use can build iceberg considering their requirements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4228682520
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -1159,6 +1201,102 @@ mod tests {
);
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ fn hdfs_native_test_storage() -> OpenDalStorage {
+ OpenDalStorage::HdfsNative(HdfsNativeStorage::new(
+ opendal::services::HdfsNativeConfig::default(),
+ OpenDalClientConfig::default(),
+ ))
+ }
+
+ /// The configuration round-trips through serde; the operator cache does
+ /// not and starts empty.
+ #[cfg(feature = "opendal-hdfs-native")]
+ #[tokio::test]
+ async fn test_hdfs_native_storage_serde_round_trip() {
Review Comment:
We should move these tests to hdfs_native mod. Also please create an issue to refactor test in this mod to separate mods.
##########
.github/workflows/ci.yml:
##########
@@ -265,6 +265,46 @@ jobs:
if: always() && matrix.test-suite.name == 'default'
run: make docker-down
+ tests-hdfs:
Review Comment:
I don't think we should add a special section in github workflow for hdfs. You should just follow other integration tests' converntion.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -1159,6 +1201,102 @@ mod tests {
);
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ fn hdfs_native_test_storage() -> OpenDalStorage {
+ OpenDalStorage::HdfsNative(HdfsNativeStorage::new(
+ opendal::services::HdfsNativeConfig::default(),
+ OpenDalClientConfig::default(),
+ ))
+ }
+
+ /// The configuration round-trips through serde; the operator cache does
+ /// not and starts empty.
+ #[cfg(feature = "opendal-hdfs-native")]
+ #[tokio::test]
+ async fn test_hdfs_native_storage_serde_round_trip() {
+ use iceberg::io::HDFS_NAME_NODE;
+
+ let props = HashMap::from([
+ (HDFS_NAME_NODE.to_string(), "hdfs://nn:8020".to_string()),
+ (
+ "hadoop.dfs.client.use.datanode.hostname".to_string(),
+ "true".to_string(),
+ ),
+ ]);
+ let storage = OpenDalStorage::HdfsNative(HdfsNativeStorage::new(
+ hdfs_native_config_parse(props).unwrap(),
+ client_config("45000").unwrap(),
+ ));
+ // Populate the cache so that its absence after the round trip means
+ // something.
+ let OpenDalStorage::HdfsNative(hdfs) = &storage else {
+ panic!("expected the HdfsNative variant");
+ };
+ hdfs_native_create_operator("hdfs:///x", &hdfs.config, &hdfs.operators)
+ .await
+ .unwrap();
+ assert!(hdfs.operators.get("hdfs://nn:8020").is_some());
+
+ let value = serde_json::to_value(&storage).unwrap();
+ assert!(value["HdfsNative"].get("operators").is_none());
+ let OpenDalStorage::HdfsNative(restored) = serde_json::from_value(value).unwrap() else {
+ panic!("expected the HdfsNative variant");
+ };
+ assert_eq!(restored.config.name_node.as_deref(), Some("hdfs://nn:8020"));
+ assert_eq!(
+ restored
+ .config
+ .options
+ .as_ref()
+ .and_then(|o| o.get("dfs.client.use.datanode.hostname")),
+ Some(&"true".to_string())
+ );
+ assert_eq!(restored.client_config.io_timeout(), Duration::from_secs(45));
+ assert!(restored.operators.get("hdfs://nn:8020").is_none());
+ }
+
+ /// `relativize_path` follows the same resolution rule as `create_operator`.
+ #[cfg(feature = "opendal-hdfs-native")]
+ #[test]
+ fn test_relativize_path_hdfs_native() {
Review Comment:
Ditto.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
Review Comment:
This should also be rename to hdfs_native
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -135,6 +144,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
I agree that we should use HdfsNative. There is also another operator in opendal which uses hdfs client's c api, and we need to reserve `Hdfs` for that.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
Review Comment:
I would suggest you to utilized the newly added derive properties macro to simplify this.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator, handle: &Handle) -> Self {
+ Self {
+ operator,
+ sentinel: RuntimeSentinel::spawn(handle),
+ }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel.alive.strong_count() > 0
+ }
+}
+
+// A poisoned lock is recovered rather than reported: writes only ever
+// replace whole entries, so a panic elsewhere cannot leave the map broken.
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Option<Operator> {
+ self.0
+ .read()
+ .unwrap_or_else(PoisonError::into_inner)
+ .get(name_node)
+ .filter(|cached| cached.runtime_alive())
+ .map(|cached| cached.operator.clone())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds; a stale entry is replaced.
+ fn insert(&self, name_node: String, op: Operator, handle: &Handle) -> Operator {
+ let mut operators = self.0.write().unwrap_or_else(PoisonError::into_inner);
+ match operators.entry(name_node) {
+ Entry::Occupied(entry) if entry.get().runtime_alive() => entry.get().operator.clone(),
+ Entry::Occupied(mut entry) => {
+ entry.insert(CachedOperator::new(op.clone(), handle));
+ op
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(CachedOperator::new(op.clone(), handle));
+ op
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) async fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &Arc<HdfsNativeConfig>,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node) {
+ return Ok((op, relative_path));
+ }
+
+ // Every operator in this crate needs a tokio runtime for its I/O (the
+ // timeout layer), so say so instead of panicking in `spawn_blocking`.
+ let handle = Handle::try_current().map_err(|_| {
+ Error::new(
+ ErrorKind::FeatureUnsupported,
+ "HDFS storage requires a tokio runtime",
+ )
+ })?;
+
+ // The build reads the Hadoop XML config synchronously, so it runs on a
+ // blocking thread and outside the lock. A racing first caller may build
+ // too; the loser is dropped before opening any connection.
+ let build_config = Arc::clone(config);
+ let build_name_node = name_node.clone();
+ let op = handle
Review Comment:
Do we still need this `spawn_blocking`? IIUC https://github.com/apache/iceberg-rust/pull/3111/changes#r4174351086 says this is not a block operation? My major concern comes from maintaining a runtime Handle using `Handle::try_current`. If we actually need this, we should consider passing the runtime in Catalog to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4194129589
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .or_else(|| hdfs_native_default_fs(config))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). The
+/// cache lives as long as the storage that owns it (clones share it) and
+/// never evicts, so the storage must not be used from another runtime once
+/// the building one is dropped: `hdfs-native` panics on a dead runtime.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // The build reads the Hadoop XML config synchronously (~0.1 ms, once
+ // per NameNode), so it runs outside the lock and is not worth a
+ // blocking-thread hop. A racing first caller may build too; the loser
+ // is dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
Review Comment:
Done this time. Since the last round the crate depends on tokio anyway (the operator cache tracks runtime liveness with it), so there was no longer a reason to keep the build inline: `create_operator` is async and the Hadoop XML read runs under `spawn_blocking`, outside the cache lock as before. The builder itself stays sync and runtime-free, which the plain unit test still pins. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6024705716
Thanks @laskoviymishka. Both blocking items and the non-blocking ones from this round are in ce78ee3..b962408: logical nameservices must be declared (plain `hdfs.name-node` serves authority-less paths only), userinfo and port 0 are rejected in path authorities, the composed `hdfs.host`/`hdfs.port` is validated, and the `HDFS_NAME_NODE` doc covers nameservices that live only in `$HADOOP_CONF_DIR`. I also tightened the test suite after a review pass: three redundant tests dropped, the warnings and the password redaction asserted directly, and fixture coverage added for authority-less paths. Description updated; full checks and the fixture suite green locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4231834993
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
Review Comment:
**Nit:** the `hdfs.host` and `hdfs.port` parsing, the `hdfs_native_nameservice` chain (L366) and the `??` at L538 are hard to follow as `Option` and `Result` combinator chains. A small helper or a `for` loop with `?` would read better.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
Review Comment:
**Should fix:** a NameNode is a `String` that can be an endpoint, a nameservice or a list, and this function re-parses the string from `hdfs_native_parse_path` to tell which. A `NameNode` newtype and an `Authority` enum would make the states explicit and let the `hdfs_native_`-prefixed free functions become methods.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
Review Comment:
**Nit:** `key[nameservice_prefix.len()..]` relies on a `starts_with` check in a separate statement, so `strip_prefix` would be safe by construction. The same applies to the slicing in `hdfs_native_redact` and `hdfs_native_parse_path`, where I do not see a panic today.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -186,6 +194,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
**Question:** adding a variant to these public enums breaks downstream exhaustive `match`es when the feature is unified on. Earlier backends did the same, so I would not block on it, but a separate PR adding `#[non_exhaustive]` may be worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4234554066
##########
crates/iceberg/src/io/storage/config/hdfs_native.rs:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
Review Comment:
Done: `hdfs.host`/`hdfs.port` work with PyIceberg's semantics (authority-less paths, port defaults to 8020), and `hdfs.user`/`hdfs.kerberos_ticket` now log a warning instead of being silently ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4231759065
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
Review Comment:
**Should fix (design):** NameNode resolution is stringly typed and re-derived on every I/O call.
`hdfs_native_parse_path` formats `hdfs://host:port` into a `String`, then this function calls `hdfs_native_name_node(&authority)` to re-parse it just to learn whether a port exists. `hdfs_native_nameservice` and `hdfs_native_default_fs` also read their state back out of opendal's `options` map on each call. That is two `Url::parse` calls plus several `format!`s before the cache lookup.
Validation is also split. The `hdfs.name-node.<ns>` sugar fails at build time, but a malformed `hadoop.dfs.ha.namenodes.*` (an id without its `rpc-address`) or `hadoop.fs.defaultFS` passes `OpenDalStorageFactory::HdfsNative.build()` and only fails at the first `exists` or `read`.
Suggestion: parse once into a small type such as `enum Authority { Endpoint(host, port), Nameservice(String), None }`, and precompute the declared nameservices and default FS in `HdfsNativeStorage::new` so bad declarations fail at build time. Moving the resolver into its own submodule would also shrink this 1347-line file.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
Review Comment:
**Question:** the cache has no eviction and gains one entry (operator, RPC client, parked sentinel task) per distinct `hdfs://host:port` seen in any path. Is it expected to stay small in practice, or should it be bounded or documented as such?
I also expect `hdfs://NN1:8020` and `hdfs://nn1:8020` to produce two entries, since `Url::host_str()` does not lowercase the host for a non-special scheme. I did not run this, so please treat it as unverified.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -372,7 +394,7 @@ impl OpenDalStorage {
/// * An [`opendal::Operator`] instance used to operate on file.
/// * Relative path to the root uri of [`opendal::Operator`].
#[allow(unreachable_code, unused_variables)]
- pub(crate) fn create_operator<'a>(
+ pub(crate) async fn create_operator<'a>(
Review Comment:
**Nit:** `create_operator` became `async` for every backend so HDFS can use `spawn_blocking`. The cost is small, but the signature change is wider than the need. Keeping it sync and doing the async build only in the `HdfsNative` arm, or lazily initialising per NameNode with a `tokio::sync::OnceCell`, would be narrower.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ let start = s
+ .find("://")
+ .filter(|&i| {
+ s[..i]
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c))
+ })
+ .map_or(0, |i| i + 3);
+ match s[start..].rfind('@') {
+ Some(at) => format!("{}***{}", &s[..start], &s[start + at..]),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| hdfs_native_invalid_path(path, reason);
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(portless) => {
+ let nameservice = portless.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
Review Comment:
**Question:** this sentinel works around the hdfs-native panic on a dead runtime inside this adapter. Liveness is only checked in `get` and `insert`, so a runtime dropped between lookup and use can still hit the panic. Could we link an upstream hdfs-native or opendal issue in a `TODO` so this can be removed once it is fixed there?
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, PoisonError, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
Review Comment:
**Nit:** attach the parse error with `.with_source(e)` instead of interpolating `{e}` into the message. The `Url` error in `hdfs_native_invalid_path` is formatted on purpose for userinfo redaction, so only this case needs changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "ryansun96 (via GitHub)" <gi...@apache.org>.
ryansun96 commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-5790230152
Would like to see this merged as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4174351086
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority — plus the relative
+/// path. Both the operator cache and `delete_stream` batching key on this,
+/// so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+impl HdfsNativeOperatorCache {
+ fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self.0.read().map_err(poisoned)?.get(name_node).cloned())
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ Ok(self
+ .0
+ .write()
+ .map_err(poisoned)?
+ .entry(name_node)
+ .or_insert(op)
+ .clone())
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // Built outside the lock: the build reads the Hadoop XML config
+ // synchronously. A racing first caller may build too; the loser is
+ // dropped before opening any connection.
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ Ok((operators.insert(name_node, op)?, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, so paths that resolve to different operators never share a
+/// deleter. Unresolvable paths key on themselves (as `hf_batch_key` does);
+/// `create_operator` then reports the real error.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
+ hdfs_native_effective_name_node(config, path)
+ .map(|(name_node, _)| name_node)
+ .unwrap_or_else(|_| path.to_string())
+}
+
+/// Build a new OpenDAL [`Operator`]: OpenDAL splits `name_node` on commas
+/// into a synthetic HA name service; `$HADOOP_CONF_DIR` XML still merges in.
+fn hdfs_native_operator_build(config: &HdfsNativeConfig, name_node: &str) -> Result<Operator> {
+ let mut cfg = config.clone();
+ cfg.name_node = Some(name_node.to_string());
+ Operator::from_config(cfg).map_err(from_opendal_error)
Review Comment:
Measured it: with `HADOOP_CONF_DIR` set, `Operator::from_config` takes ~0.1 ms (p90 116 µs, N=200), once per NameNode per storage thanks to the cache; a `spawn_blocking` round trip alone is ~10 µs. Nothing else runs at build, since connections are lazy and Kerberos setup is a no-op unless configured. The other backends here do comparable sync I/O on every `create_operator` call (HF reads the token file, fs stats the root), so I'd rather not add a tokio-specific offload for this one. I documented the cost at the build site instead; it will be in the next push together with the other fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4174553646
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
Review Comment:
A bare `host:port` actually works: that is Hadoop's own rpc-address format, opendal only strips an optional `hdfs://` per entry (its tests mix both spellings), and I verified hdfs-native dials it. So I'd rather not reject it. The misconfiguration that does fail opaquely is an entry without a port (`hdfs://ns1`), which hdfs-native can only report as "invalid socket address" at the first I/O. Each entry is now validated as `host:port` at build time, with an error naming the property and the entry. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4196324978
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
+ name_node
+ .rsplit_once(':')
+ .is_some_and(|(_, port)| port.parse::<u16>().is_ok())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // Each entry is dialed as `host:port` (`hdfs://` optional); a portless
+ // one would fail only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| !hdfs_native_has_port(entry)) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_has_port(&authority) => authority,
+ Some(logical) => config.name_node.clone().ok_or_else(|| {
+ let logical = logical.trim_start_matches("hdfs://");
+ invalid(format!(
+ "logical nameservice `{logical}` requires `{HDFS_NAME_NODE}`"
+ ))
+ })?,
+ None => match config
+ .name_node
+ .clone()
+ .or_else(|| hdfs_native_default_fs(config))
+ {
+ Some(name_node) if hdfs_native_has_port(&name_node) => name_node,
+ Some(logical) => {
+ return Err(invalid(format!(
+ "`{FS_DEFAULT_FS}` {logical} has no port, a logical nameservice requires `{HDFS_NAME_NODE}`"
+ )));
+ }
+ None => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). An
+/// entry is rebuilt once that runtime is gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ /// `None` when built outside any runtime.
+ sentinel: Option<RuntimeSentinel>,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator) -> Self {
+ let sentinel = Handle::try_current()
+ .ok()
+ .map(|handle| RuntimeSentinel::spawn(&handle));
+ Self { operator, sentinel }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel
+ .as_ref()
+ .is_none_or(|sentinel| sentinel.alive.strong_count() > 0)
+ }
+}
+
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self
+ .0
+ .read()
+ .map_err(poisoned)?
+ .get(name_node)
+ .filter(|cached| cached.runtime_alive())
+ .map(|cached| cached.operator.clone()))
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds; a stale entry is replaced.
+ fn insert(&self, name_node: String, op: Operator) -> Result<Operator> {
+ let mut operators = self.0.write().map_err(poisoned)?;
+ match operators.entry(name_node) {
+ Entry::Occupied(entry) if entry.get().runtime_alive() => {
+ Ok(entry.get().operator.clone())
+ }
+ Entry::Occupied(mut entry) => {
+ entry.insert(CachedOperator::new(op.clone()));
+ Ok(op)
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(CachedOperator::new(op.clone()));
+ Ok(op)
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+fn poisoned<T>(_: T) -> Error {
+ Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned")
+}
+
+/// Creates an operator for the path, reusing the cached one for its
+/// effective NameNode.
+pub(crate) async fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &Arc<HdfsNativeConfig>,
+ operators: &HdfsNativeOperatorCache,
+) -> Result<(Operator, &'a str)> {
+ let (name_node, relative_path) = hdfs_native_effective_name_node(config, path)?;
+
+ if let Some(op) = operators.get(&name_node)? {
+ return Ok((op, relative_path));
+ }
+
+ // The build reads the Hadoop XML config synchronously, so it runs on a
+ // blocking thread and outside the lock. A racing first caller may build
+ // too; the loser is dropped before opening any connection.
+ let build_config = Arc::clone(config);
+ let build_name_node = name_node.clone();
+ let op = tokio::task::spawn_blocking(move || {
Review Comment:
I'd gate this on `Handle::try_current()` — use `spawn_blocking` when there's a runtime, otherwise build inline (or return `FeatureUnsupported`) — because `spawn_blocking` panics outside a tokio runtime, where the rest of the crate stays executor-agnostic. So this fixes the executor-blocking I flagged last round but swaps it for a hard crash under `futures::executor::block_on` / smol, which is the kind of panic the crate otherwise avoids.
It also leaves dead code: `CachedOperator::new`'s `sentinel: None` arm and the "a private one when built outside any" comment can't be reached on this path anymore — only the direct-insert test hits them. Worth dropping that branch if it's genuinely unreachable.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
+ name_node
+ .rsplit_once(':')
+ .is_some_and(|(_, port)| port.parse::<u16>().is_ok())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // Each entry is dialed as `host:port` (`hdfs://` optional); a portless
+ // one would fail only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| !hdfs_native_has_port(entry)) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_has_port(&authority) => authority,
+ Some(logical) => config.name_node.clone().ok_or_else(|| {
Review Comment:
The sharp edge here is `delete_prefix`/`delete_stream`: with `hdfs.name-node` set, every portless authority resolves to that one configured list, so `hdfs://ns-a/x`, `hdfs://ns-b/x`, and a typo'd authority all route to the same cluster — an orphan-cleanup pass keyed on the wrong name deletes against the wrong cluster. Java would just fail on an unknown nameservice. The flip side is that a nameservice defined only in `$HADOOP_CONF_DIR` is rejected here even though the client could resolve it from XML.
Both come down to whether opendal 0.58.1 accepts a bare logical `name_node` and resolves it from the Hadoop XML — I couldn't confirm that from the registry sources. If it does, I'd forward a portless authority through unchanged and let the client resolve it; if it doesn't, I'd key the override per-nameservice and document the single-cluster assumption loudly. Do you know how opendal treats a logical `name_node` here?
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,928 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+
+/// `hdfs-native` dials a NameNode as a socket address and has no default
+/// port, so anything without one can only be a logical nameservice name.
+fn hdfs_native_has_port(name_node: &str) -> bool {
Review Comment:
This only looks after the last `:` and takes any `u16`, so it waves through a few shapes that aren't a NameNode: `viewfs://x:1`, `foo://x:1`, `nn:0`, `nn:+80` (`parse` accepts the leading `+`), and bare `::1` reads as port 1. Separately, `nn:8020` and `hdfs://nn:8020` normalize to different cache keys for the same cluster. I'd strip an optional `hdfs://`, reject any other `scheme://`, and normalize before joining — tightens the validation and dedupes the keys in one pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4200043240
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
+ let nameservice = logical.trim_start_matches("hdfs://");
+ match hdfs_native_nameservice(config, nameservice)? {
+ Some(name_node) => name_node,
+ None => config.name_node.clone().ok_or_else(|| {
Review Comment:
Agreed, guessing was the wrong default. The plain key now serves authority-less paths only, a logical nameservice resolves only through its declaration (`hdfs.name-node.<ns>`, or Hadoop's own keys via `hadoop.*`), and an undeclared name is a `DataInvalid` naming the key to set, before any I/O. Verified against two local NameNodes that a typo'd name and a bare second-cluster hostname are rejected while declared and authority-less paths reach the right one. The `HDFS_NAME_NODE` doc also notes that nameservices living only in `$HADOOP_CONF_DIR` must be declared in properties.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
Review Comment:
Both confirmed and rejected in `hdfs_native_parse_path` with `DataInvalid`: userinfo (the message names the host only, so a password never lands in a log) and port 0 ("cannot be dialed"). An authority with a port is now valid by construction, so nothing with a port can fall into the nameservice branch anymore; test added.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
Review Comment:
Confirmed, including a third shape (`hdfs.host=hdfs://nn` became `hdfs://[hdfs://nn]:8020`). The composed value now runs through the same normalizer as every other NameNode spelling, so a host carrying a scheme, a port, or a path, and port 0, fail at config time with an error naming `hdfs.host`/`hdfs.port`; tests added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4216882220
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ // Not echoing the path: it may carry a password.
+ let host = url.host_str().unwrap_or_default();
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path for host `{host}`: userinfo is not supported"),
+ ));
+ }
+ if url.port() == Some(0) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, port 0 cannot be dialed"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through its declaration, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
Review Comment:
Checked this end to end. hdfs-native on its own does resolve a `HADOOP_CONF_DIR` nameservice, but opendal never hands it the authority: since apache/opendal#7248 (0.56) it builds the client as `hdfs://nameservice` and writes each `name_node` entry as `dfs.namenode.rpc-address.nameservice.nnN`. So `hdfs://ns1` passed through, with `ns1` defined in hdfs-site.xml, becomes an rpc-address without a port, and so does a portless host: neither is ever dialed (a listener on 8020 saw nothing), and hdfs-native retries `invalid socket address` through its default 15 failovers with backoff before giving up. hdfs-native has no default port either; given `hdfs://127.0.0.1` directly it returns `No NameNode hosts found`. So pass-through would turn today's immediate error into a slow, opaque one.
Kept strict for now. The `HDFS_NAME_NODE` doc no longer says "as in Hadoop" and states the divergence, the error now reads "`ns1` has no port and is not a declared nameservice; add the port or set `hdfs.name-node.ns1`", and the README notes the limitation. The real fix is upstream, letting opendal take a logical nameservice as `name_node` and pass it as the client URL; once that lands, undeclared names can pass through as you suggest.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
Review Comment:
Fixed, and the leak was wider than these two sites: the `hdfs.name-node`, `hdfs.name-node.<nameservice>`, `hdfs.host` and `fs.defaultFS` errors echoed their values too. Every `Invalid hdfs path` error now goes through one helper that masks everything between the scheme and the last `@` (`hdfs://***@nn:bad/x`), so the path keeps its context and a password with a raw `/` or `#` is covered as well; the config errors use the same masking. A test covers each of these errors and asserts none contains the user or the password. `OpenDalResolvingStorage`'s scheme lookup echoes a bad path the same way for every scheme, which predates this PR; worth a separate fix.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ mut m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.clone()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {host}:{port}, expected a host name or IP and a port"
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ None if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ // Not echoing the path: it may carry a password.
+ let host = url.host_str().unwrap_or_default();
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path for host `{host}`: userinfo is not supported"),
+ ));
+ }
+ if url.port() == Some(0) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, port 0 cannot be dialed"),
+ ));
+ }
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through its declaration, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
+ let nameservice = logical.trim_start_matches("hdfs://");
+ hdfs_native_nameservice(config, nameservice)?.ok_or_else(|| {
+ invalid(format!(
+ "logical nameservice `{nameservice}` is not declared; set `{HDFS_NAME_NODE}.{nameservice}`"
+ ))
+ })?
+ }
+ None => match (&config.name_node, hdfs_native_default_fs(config)) {
+ (Some(name_node), _) => name_node.clone(),
+ (None, Some(default_fs)) => hdfs_native_name_node(default_fs).ok_or_else(|| {
+ invalid(format!(
+ "`{FS_DEFAULT_FS}` {default_fs} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`"
+ ))
+ })?,
+ (None, None) => {
+ return Err(invalid(format!(
+ "authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ )));
+ }
+ },
+ };
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, as written.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<&str> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let name_nodes = ids
+ .split(',')
+ .map(str::trim)
+ .filter(|id| !id.is_empty())
+ .map(|id| {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built. An entry is rebuilt once that runtime is
+/// gone, as `hdfs-native` panics when
+/// it spawns onto a dead one. The cache lives as long as the storage that
+/// owns it (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, CachedOperator>>>);
+
+#[derive(Debug)]
+struct CachedOperator {
+ operator: Operator,
+ sentinel: RuntimeSentinel,
+}
+
+/// A task parked on the building runtime that owns the token, so the token
+/// outlives it only while that runtime is alive. Aborted on drop so entries
+/// do not leave parked tasks behind.
+#[derive(Debug)]
+struct RuntimeSentinel {
+ alive: Weak<()>,
+ task: JoinHandle<()>,
+}
+
+impl RuntimeSentinel {
+ fn spawn(handle: &Handle) -> Self {
+ let token = Arc::new(());
+ let alive = Arc::downgrade(&token);
+ let task = handle.spawn(async move {
+ let _token = token;
+ std::future::pending::<()>().await
+ });
+ Self { alive, task }
+ }
+}
+
+impl Drop for RuntimeSentinel {
+ fn drop(&mut self) {
+ self.task.abort();
+ }
+}
+
+impl CachedOperator {
+ fn new(operator: Operator, handle: &Handle) -> Self {
+ Self {
+ operator,
+ sentinel: RuntimeSentinel::spawn(handle),
+ }
+ }
+
+ fn runtime_alive(&self) -> bool {
+ self.sentinel.alive.strong_count() > 0
+ }
+}
+
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Result<Option<Operator>> {
+ Ok(self
+ .0
+ .read()
+ .map_err(poisoned)?
Review Comment:
Done: both sites recover the guard with `PoisonError::into_inner`, `get`/`insert` return plain values, and the error helper is gone. Recovery is safe since writes only ever replace whole entries. A test poisons the lock and checks the cache still builds and serves operators.
##########
crates/storage/opendal/tests/hdfs_native_runtime_test.rs:
##########
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A cached HDFS operator is bound to the tokio runtime that built it; once
+//! that runtime is gone it must be rebuilt rather than reused. Needs no
+//! HDFS: the NameNode is a local listener that only counts dials.
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::net::TcpListener;
+ use std::sync::Arc;
+ use std::sync::atomic::{AtomicUsize, Ordering};
+ use std::time::Duration;
+
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::OpenDalStorageFactory;
+
+ /// Accepts and immediately closes connections, counting them.
+ fn fake_name_node() -> (u16, Arc<AtomicUsize>) {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let port = listener.local_addr().unwrap().port();
+ let dials = Arc::new(AtomicUsize::new(0));
+ let counter = dials.clone();
+ std::thread::spawn(move || {
+ for stream in listener.incoming() {
+ let Ok(_stream) = stream else { break };
+ counter.fetch_add(1, Ordering::SeqCst);
+ }
+ });
+ (port, dials)
+ }
+
+ fn stat(runtime: &tokio::runtime::Runtime, file_io: &FileIO) {
+ runtime.block_on(async {
+ let input = file_io.new_input("hdfs:///f").unwrap();
+ // The fake NameNode never answers, so this fails; only the dial matters.
+ let _ = tokio::time::timeout(Duration::from_secs(10), input.metadata()).await;
+ });
+ }
+
+ #[test]
+ fn test_hdfs_operator_is_rebuilt_after_its_runtime_is_dropped() {
+ let (port, dials) = fake_name_node();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, format!("hdfs://127.0.0.1:{port}"))
+ .with_prop("hadoop.dfs.client.failover.max.attempts", "1")
+ .build();
+
+ let first = tokio::runtime::Runtime::new().unwrap();
+ stat(&first, &file_io);
+ let after_first = dials.load(Ordering::SeqCst);
+ assert!(after_first >= 1, "the NameNode was never dialed");
+ drop(first);
+
+ // Reusing the FileIO from another runtime used to panic inside
+ // hdfs-native, whose client was bound to the dropped runtime.
+ let second = tokio::runtime::Runtime::new().unwrap();
+ stat(&second, &file_io);
+ assert!(
+ dials.load(Ordering::SeqCst) > after_first,
Review Comment:
Fixed the sampling. One correction on the premise: the first stat never reaches the 10s timeout. It fails in about 1.5s after up to 4 dials, and over 30 runs no dial arrived after the sample. A reused stale operator also panics inside hdfs-native before the assertion, as the original bug did. Still, the order was wrong in principle, so the test now drops the first runtime, opens a marker connection and waits for the listener to close it. The listener accepts in order, so every earlier dial is counted before `after_first` is sampled, with no sleeps.
The cache is crate-private, so the identity check went into the runtime-shutdown unit test: it asserts the rebuilt operator is a different instance from the stale one (`Operator::into_parts` and `Arc::ptr_eq`). A mutation that refreshes the sentinel but keeps the old operator fails only that assertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4035433608
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the `hdfs-namenode`/`hdfs-datanode` services from
+//! `dev/docker-compose.yaml` (started by `make docker-up`); the fixture
+//! uses host networking, which needs Linux or a recent Docker runtime.
+
+#[cfg(feature = "opendal-hdfs-native")]
Review Comment:
You're right — that paragraph was left over from before the `#[ignore]` step was removed at @blackmwk's request, and the counts were stale. Fixed in d17ae38 by following the HF tests' approach: the HDFS tests skip unless `ICEBERG_TEST_HDFS_ENDPOINT` is set, the compose services sit behind a `hdfs` profile, and the Linux CI job opts in via env. `make test` now skips them on any platform. Description updated.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
+ Some(name_node) => name_node,
+ None => {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ ));
+ }
+ };
+
+ // Fast path: check read lock first.
+ {
+ let cache = operators
+ .read()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ if let Some(op) = cache.get(&name_node) {
+ return Ok((op.clone(), relative_path));
+ }
+ }
+
+ // Slow path: build and insert under write lock, re-checking for a
+ // concurrent insert.
+ let mut cache = operators
+ .write()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ let op = match cache.get(&name_node) {
+ Some(op) => op.clone(),
+ None => {
+ let op = hdfs_native_operator_build(config, &name_node)?;
Review Comment:
Agreed, fixed in d17ae38: the operator is built outside the lock and inserted with `entry().or_insert`, so a racing first caller may build a duplicate that gets dropped before any I/O. One note on scope: `NameServiceProxy::new` doesn't connect (connections are lazy on first RPC), so the work under the old lock was two local XML reads per NameNode — still not worth holding a write lock for.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +262,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: Arc<RwLock<HashMap<String, Operator>>>,
Review Comment:
Fair point. d17ae38 wraps the cache in `HdfsNativeOperatorCache` (private field, `Default`), so the public API only names that type and the internal representation can change freely.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
+ Some(name_node) => name_node,
+ None => {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require the `{HDFS_NAME_NODE}` property"
+ ),
+ ));
+ }
+ };
+
+ // Fast path: check read lock first.
+ {
+ let cache = operators
+ .read()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ if let Some(op) = cache.get(&name_node) {
+ return Ok((op.clone(), relative_path));
+ }
+ }
+
+ // Slow path: build and insert under write lock, re-checking for a
+ // concurrent insert.
+ let mut cache = operators
+ .write()
+ .map_err(|_| Error::new(ErrorKind::Unexpected, "HDFS operator cache lock poisoned"))?;
+ let op = match cache.get(&name_node) {
+ Some(op) => op.clone(),
+ None => {
+ let op = hdfs_native_operator_build(config, &name_node)?;
+ cache.insert(name_node, op.clone());
+ op
+ }
+ };
+
+ Ok((op, relative_path))
+}
+
+/// Returns the `delete_stream` grouping key for a path: the effective
+/// NameNode, mirroring the operator-cache key so paths that resolve to
+/// different operators never share a deleter.
+pub(crate) fn hdfs_native_batch_key(config: &HdfsNativeConfig, path: &str) -> String {
Review Comment:
Done in d17ae38: both `create_operator` and the batch key go through one `hdfs_native_effective_name_node`. Unresolvable paths now key on the full path (same as `hf_batch_key`) instead of `""`. Dropped the duplicate precedence test.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,350 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::RwLock;
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Creates an operator for the path, cached per effective NameNode (the
+/// configured `hdfs.name-node`, else the path authority) — each operator
+/// holds an HDFS client with live RPC connections.
+pub(crate) fn hdfs_native_create_operator<'a>(
+ path: &'a str,
+ config: &HdfsNativeConfig,
+ operators: &RwLock<HashMap<String, Operator>>,
+) -> Result<(Operator, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+
+ let name_node = match config.name_node.clone().or(authority_name_node) {
Review Comment:
Good catch — `from_config` does bypass the builder's empty-string guard. d17ae38 trims the property, strips a trailing `/`, and treats an empty value as unset, with tests for both cases.
##########
dev/docker-compose.yaml:
##########
@@ -147,6 +147,49 @@ services:
timeout: 5s
retries: 5
+ # =============================================================================
+ # HDFS - single-node NameNode + DataNode for HDFS tests
+ # =============================================================================
+ # hdfs-native connects to DataNodes by their registered IP — unroutable on
+ # a docker bridge, hence host networking (needs Linux or a recent runtime).
+ hdfs-namenode:
+ image: apache/hadoop:3.5.0
+ network_mode: "host"
+ command: ["hdfs", "namenode"]
+ environment:
+ ENSURE_NAMENODE_DIR: "/tmp/hadoop-root/dfs/name"
+ extra_hosts:
+ - "docker-desktop:127.0.0.1"
+ volumes:
+ - ./hdfs/core-site.xml:/opt/hadoop/etc/hadoop/core-site.xml:ro
+ - ./hdfs/hdfs-site.xml:/opt/hadoop/etc/hadoop/hdfs-site.xml:ro
+ healthcheck:
+ test: ["CMD-SHELL", "hdfs dfsadmin -safemode get | grep -q OFF"]
+ interval: 5s
+ timeout: 15s
+ retries: 30
+ start_period: 30s
+
+ hdfs-datanode:
Review Comment:
It's possible: with host networking, each extra DataNode needs its own `dfs.datanode.{address,http.address,ipc.address}` ports and data dir. I kept the fixture single-node to match opendal's own HDFS fixture — DataNode count doesn't change the FileIO adapter's behavior (failover and erasure-coded reads are hdfs-native's domain), and it's already the slowest service in `make docker-up`. Happy to add a two-DataNode variant as a follow-up if there's a scenario you'd like covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4348503723
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, LazyLock, PoisonError, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use iceberg_property_macro::Properties;
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Runtime;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| hdfs_native_endpoint(host, port))
+}
+
+/// An endpoint as every source spells it, the host lowercased because host
+/// names are case-insensitive: `NN1:8020` and `nn1:8020` share one client,
+/// as they share one `FileSystem` in Hadoop's cache.
+fn hdfs_native_endpoint(host: &str, port: u16) -> String {
+ format!("hdfs://{}:{port}", host.to_ascii_lowercase())
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {}, expected host:port (hdfs:// optional)",
+ hdfs_native_redact(entry)
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Prefix of `hdfs.name-node.<nameservice>`, which declares one HA nameservice.
+const HDFS_NAME_NODE_DECLARATION_PREFIX: &str = "hdfs.name-node.";
+
+/// The properties this backend reads; unknown keys are ignored. Blank values
+/// count as unset, as in PyIceberg.
+#[derive(Properties)]
+struct HdfsNativeProperties {
+ /// `hdfs.name-node`, each entry normalized.
+ #[property(
+ key = HDFS_NAME_NODE,
+ default = Vec::new(),
+ parse_with = parse_name_node_list
+ )]
+ name_node: Vec<String>,
+ /// `hdfs.name-node.<nameservice>` declarations, keyed by nameservice.
+ #[property(prefix = HDFS_NAME_NODE_DECLARATION_PREFIX)]
+ nameservices: HashMap<String, String>,
+ #[property(key = HDFS_HOST, default = "", parse_with = parse_trimmed)]
+ host: String,
+ #[property(key = HDFS_PORT, default = "", parse_with = parse_trimmed)]
+ port: String,
+ /// `hadoop.*` keys with the prefix stripped, forwarded to the client.
+ #[property(prefix = HDFS_HADOOP_CONF_PREFIX)]
+ hadoop: HashMap<String, String>,
+}
+
+// Entries are trimmed one by one: opendal splits the list on `,` as is, so a
+// space after a comma would break failover to that NameNode. An empty list
+// stays unset because `Operator::from_config` bypasses the builder's
+// empty-string guard and `Some("")` would shadow the path-authority fallback.
+fn parse_name_node_list(value: &str) -> Result<Vec<String>> {
+ hdfs_native_name_node_list(HDFS_NAME_NODE, value)
+}
+
+fn parse_trimmed(value: &str) -> Result<String> {
+ Ok(value.trim().to_string())
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`]; dropped properties are
+/// logged as warnings.
+pub(crate) fn hdfs_native_config_parse(m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ hdfs_native_config_parse_with(m, |warning| tracing::warn!("{warning}"))
+}
+
+/// The parser proper, with the warning sink injected so tests can see what
+/// was dropped without a tracing subscriber.
+fn hdfs_native_config_parse_with(
+ m: HashMap<String, String>,
+ mut warn: impl FnMut(String),
+) -> Result<HdfsNativeConfig> {
+ let props = HdfsNativeProperties::from_properties(&m)?;
+ let mut options = props.hadoop;
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, value) in props.nameservices {
+ let key = format!("{HDFS_NAME_NODE_DECLARATION_PREFIX}{nameservice}");
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{HDFS_NAME_NODE_DECLARATION_PREFIX}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.contains_key(key) {
+ warn(format!(
+ "`{key}` is not supported by the hdfs-native backend and is ignored"
+ ));
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let port = match props.port.as_str() {
+ "" => None,
+ port => Some(port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}"),
+ )
+ .with_source(e)
+ })?),
+ };
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match props.host.as_str() {
+ "" if port.is_some() => {
+ warn(format!(
+ "`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored"
+ ));
+ }
+ "" => {}
+ host => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let bracketed = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host.to_string()
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ // Validated like every other NameNode spelling, so a host that
+ // carries a scheme or port fails here and not at the first I/O.
+ let default_fs = hdfs_native_name_node(&format!("hdfs://{bracketed}:{port}"))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{HDFS_HOST}`/`{HDFS_PORT}`: {}, expected a host name or IP and a port",
+ hdfs_native_redact(&format!("{host}:{port}"))
+ ),
+ )
+ })?;
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert(default_fs);
+ }
+ }
+
+ let mut cfg = HdfsNativeConfig::default();
+ if !props.name_node.is_empty() {
+ cfg.name_node = Some(props.name_node.join(","));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ // Every declaration, sugar or passed through, and the default filesystem
+ // fail here rather than at the first path that uses them.
+ for nameservice in cfg.options.iter().flat_map(|options| {
+ options.keys().filter_map(|key| {
+ key.strip_prefix(HA_NAMENODES_PREFIX)
+ .and_then(|rest| rest.strip_prefix('.'))
+ })
+ }) {
+ hdfs_native_nameservice(&cfg, nameservice)?;
+ }
+ hdfs_native_default_name_node(&cfg)?;
+ Ok(cfg)
+}
+
+/// `s` with everything between its scheme and its last `@` masked, so an
+/// error message never echoes userinfo, even a password holding a raw `/`,
+/// `?`, `#` or `://` that a URL parser would split elsewhere. An `@` further
+/// along the path is masked the same way, which only costs context.
+fn hdfs_native_redact(s: &str) -> String {
+ // Only a real scheme is kept; anything else before `://` may be userinfo.
+ let (scheme, rest) = match s.split_once("://") {
+ Some((scheme, rest))
+ if scheme
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || "+-.".contains(c)) =>
+ {
+ (format!("{scheme}://"), rest)
+ }
+ _ => (String::new(), s),
+ };
+ match rest.rsplit_once('@') {
+ Some((_, after)) => format!("{scheme}***@{after}"),
+ None => s.to_string(),
+ }
+}
+
+/// A `DataInvalid` error for an HDFS path, its userinfo masked.
+fn hdfs_native_invalid_path(path: &str, reason: impl std::fmt::Display) -> Error {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {}, {reason}", hdfs_native_redact(path)),
+ )
+}
+
+/// A path's authority, as the resolver reads it.
+#[derive(Debug, PartialEq)]
+pub(crate) enum HdfsNativeAuthority {
+ /// `hdfs://host:port`, normalized: dialed as is.
+ NameNode(String),
+ /// `hdfs://name` without a port: a nameservice that must be declared.
+ Nameservice(String),
+ /// `hdfs:///path`: served by `hdfs.name-node`, else `fs.defaultFS`.
+ Default,
+}
+
+/// Parse an HDFS path into its authority and the relative path (no leading
+/// `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(HdfsNativeAuthority, &str)> {
+ let url = Url::parse(path).map_err(|e| hdfs_native_invalid_path(path, e))?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(hdfs_native_invalid_path(path, "expected scheme `hdfs://`"));
+ };
+ // Userinfo has nowhere to go and port 0 cannot be dialed; silently
+ // dropping the one or reading the other as a logical name would mislead.
+ if !url.username().is_empty() || url.password().is_some() {
+ return Err(hdfs_native_invalid_path(path, "userinfo is not supported"));
+ }
+ if url.port() == Some(0) {
+ return Err(hdfs_native_invalid_path(path, "port 0 cannot be dialed"));
+ }
+
+ let authority = match (url.host_str().filter(|h| !h.is_empty()), url.port()) {
+ (Some(host), Some(port)) => HdfsNativeAuthority::NameNode(hdfs_native_endpoint(host, port)),
+ (Some(host), None) => HdfsNativeAuthority::Nameservice(host.to_string()),
+ (None, _) => HdfsNativeAuthority::Default,
+ };
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Take the path from the original input instead: everything
+ // after the first `/` following the `hdfs://` prefix. Opendal paths must
+ // not start with `/` (`Deleter::delete` rejects them).
+ let rel = after_scheme
+ .split_once('/')
+ .map_or("", |(_, path)| path.trim_start_matches('/'));
+
+ Ok((authority, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. An
+/// authority with a port is used as is, a portless one must be a declared
+/// nameservice, and an authority-less path uses `hdfs.name-node`, else
+/// `fs.defaultFS`. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = match authority {
+ HdfsNativeAuthority::NameNode(name_node) => name_node,
+ HdfsNativeAuthority::Nameservice(nameservice) => {
+ hdfs_native_nameservice(config, &nameservice)?.ok_or_else(|| {
+ hdfs_native_invalid_path(
+ path,
+ format!(
+ "`{nameservice}` has no port and is not a declared nameservice; add the port or set `{HDFS_NAME_NODE}.{nameservice}`"
+ ),
+ )
+ })?
+ }
+ HdfsNativeAuthority::Default => hdfs_native_default_name_node(config)?.ok_or_else(|| {
+ hdfs_native_invalid_path(
+ path,
+ format!("authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"),
+ )
+ })?,
+ };
+ Ok((name_node, relative_path))
+}
+
+/// The NameNode that serves authority-less paths: `hdfs.name-node`, else
+/// `fs.defaultFS`, which must then be an HDFS `host:port`.
+fn hdfs_native_default_name_node(config: &HdfsNativeConfig) -> Result<Option<String>> {
+ if let Some(name_node) = &config.name_node {
+ return Ok(Some(name_node.clone()));
+ }
+ let Some(default_fs) = config
+ .options
+ .as_ref()
+ .and_then(|options| options.get(FS_DEFAULT_FS))
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+ else {
+ return Ok(None);
+ };
+ hdfs_native_name_node(default_fs).map(Some).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{FS_DEFAULT_FS}`: {} is not an HDFS host:port, a logical nameservice requires `{HDFS_NAME_NODE}`",
+ hdfs_native_redact(default_fs)
+ ),
+ )
+ })
+}
+
+/// The NameNodes that Hadoop's keys in the forwarded options declare for a
+/// nameservice, if any. A declaration with a missing or malformed address is
+/// an error rather than a silent fallback.
+fn hdfs_native_nameservice(config: &HdfsNativeConfig, nameservice: &str) -> Result<Option<String>> {
+ let Some(options) = config.options.as_ref() else {
+ return Ok(None);
+ };
+ let Some(ids) = options.get(&format!("{HA_NAMENODES_PREFIX}.{nameservice}")) else {
+ return Ok(None);
+ };
+ let mut name_nodes = Vec::new();
+ for id in ids.split(',').map(str::trim).filter(|id| !id.is_empty()) {
+ let key = format!("{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}");
+ let Some(name_node) = options
+ .get(&key)
+ .and_then(|value| hdfs_native_name_node(value))
+ else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Nameservice `{nameservice}` declares NameNode `{id}` but `{key}` is missing or not host:port"
+ ),
+ ));
+ };
+ name_nodes.push(name_node);
+ }
+ if name_nodes.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Nameservice `{nameservice}` declares no NameNodes"),
+ ));
+ }
+ Ok(Some(name_nodes.join(",")))
+}
+
+/// State of [`OpenDalStorage::HdfsNative`](crate::OpenDalStorage::HdfsNative):
+/// the parsed configuration and the per-NameNode operator cache. Only the
+/// storage factories build it.
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct HdfsNativeStorage {
+ pub(crate) config: Arc<HdfsNativeConfig>,
+ #[serde(skip, default)]
+ pub(crate) operators: HdfsNativeOperatorCache,
+ #[serde(default)]
+ pub(crate) client_config: OpenDalClientConfig,
+}
+
+impl HdfsNativeStorage {
+ pub(crate) fn new(config: HdfsNativeConfig, client_config: OpenDalClientConfig) -> Self {
+ Self {
+ config: Arc::new(config),
+ operators: HdfsNativeOperatorCache::default(),
+ client_config,
+ }
+ }
+}
+
+/// Operators cached per effective NameNode: one per endpoint or nameservice
+/// the storage's paths reach, which in practice is a handful of clusters, so
+/// it is not bounded. The cache lives as long as the storage that owns it
+/// (clones share it).
+#[derive(Clone, Debug, Default)]
+pub(crate) struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
+
+// A poisoned lock is recovered rather than reported: writes only insert
+// whole entries, so a panic elsewhere cannot leave the map broken.
+impl HdfsNativeOperatorCache {
+ pub(crate) fn get(&self, name_node: &str) -> Option<Operator> {
+ self.0
+ .read()
+ .unwrap_or_else(PoisonError::into_inner)
+ .get(name_node)
+ .cloned()
+ }
+
+ /// Inserts `op` unless a concurrent caller got there first, returning
+ /// whichever operator the cache now holds.
+ fn insert(&self, name_node: String, op: Operator) -> Operator {
+ self.0
+ .write()
+ .unwrap_or_else(PoisonError::into_inner)
+ .entry(name_node)
+ .or_insert(op)
+ .clone()
+ }
+
+ #[cfg(test)]
+ fn len(&self) -> usize {
+ self.0.read().unwrap().len()
+ }
+}
+
+/// The runtime `hdfs-native` clients are built in. A client spawns its
+/// NameNode connections on the runtime current when it is built and panics
+/// once that runtime is gone, so building it here keeps a cached operator
+/// usable from any caller's runtime. Started on first use.
+// TODO: pass a runtime to the client instead of entering this one once
+// opendal's hdfs-native service exposes hdfs-native's
+// `ClientBuilder::with_io_runtime` (Kimahriman/hdfs-native#255).
+static HDFS_NATIVE_RUNTIME: LazyLock<Runtime> = LazyLock::new(|| {
Review Comment:
Opened #3509. #1945 and the closed #2602 cover nearby ground (running OpenDAL work on a configured runtime); this one is about storage factories receiving the catalog's `Runtime`.
##########
crates/storage/opendal/README.md:
##########
@@ -82,3 +85,40 @@ async fn main() -> iceberg::Result<()> {
Ok(())
}
```
+
+## HDFS
Review Comment:
Removed, and opened #3510 to document every backend, with the HDFS text from this PR linked as a starting point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r3892676270
##########
.github/workflows/ci.yml:
##########
@@ -232,6 +232,16 @@ jobs:
cargo test --no-fail-fast ${{ matrix.test-suite.args }}
fi
+ # HDFS tests are `#[ignore]`d (docker fixture needs Linux host
+ # networking); opt them in here by test-name filter.
+ - name: Run HDFS integration tests
Review Comment:
I don't think we need a special case to discover hdfs tests suite, using rust's builtin it discover usage would be enough.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
I think you could use the newly added `derive[Propertes]` for this config.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+use std::collections::HashMap;
+
+use serde::{Deserialize, Serialize};
+use typed_builder::TypedBuilder;
+
+use super::StorageConfig;
+use crate::Result;
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
+
+/// HDFS storage configuration.
+///
+/// This struct contains all the configuration options for connecting to HDFS.
+/// Use the builder pattern via `HdfsConfig::builder()` to construct instances.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, TypedBuilder)]
+pub struct HdfsConfig {
Review Comment:
```suggestion
pub(crate) struct HdfsConfig {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4174790837
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
+ .filter(|s| !s.is_empty())
+ {
+ cfg.name_node = Some(name_node);
+ }
+
+ let options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
Review Comment:
Checked this: `Url::host_str()` returns the host slice of the serialized URL, so IPv6 keeps its brackets. `hdfs://[::1]:8020/a/b` already resolves to `hdfs://[::1]:8020` on the current code; I added `test_hdfs_native_parse_path_ipv6_authority_keeps_brackets` to pin it. I also pointed the client at a local `[::1]` listener through the path authority and it connects fine. `url.host()` would give the same string, so I left the code as is.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -460,6 +499,11 @@ impl OpenDalStorage {
))
}
}
+ #[cfg(feature = "opendal-hdfs-native")]
+ OpenDalStorage::HdfsNative { .. } => {
+ let (_, relative_path) = hdfs_native_parse_path(path)?;
Review Comment:
Done: `relativize_path` now goes through `hdfs_native_effective_name_node`, so an authority-less path errors the same way in both unless a NameNode is configured; test updated. For the record it was not reachable (`delete_stream` only relativizes paths whose batch deleter `create_operator` already built), but one rule for all callers is cleaner.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -243,6 +261,18 @@ pub enum OpenDalStorage {
/// GCS configuration.
config: Arc<GcsConfig>,
},
+ /// HDFS storage variant.
+ ///
+ /// The NameNode is taken from the `hdfs.name-node` property when set
+ /// (comma-separated endpoints enable HA failover), else the path authority.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative {
+ /// HDFS configuration.
+ config: Arc<HdfsNativeConfig>,
+ /// Operator cache keyed by effective NameNode.
+ #[serde(skip, default)]
+ operators: HdfsNativeOperatorCache,
Review Comment:
The representation is already private: `public-api.txt` only shows `HdfsNativeOperatorCache(_)` with Clone/Default/Debug, so swapping the internals is not a breaking change. Enum variant fields are always public in Rust, so a cached variant has to expose some opaque type either way; `HdfsNative(Arc<HdfsNativeState>)` would just rename it and hide `config`, which every other variant exposes. `Memory(Operator)` and `S3::customized_credential_load` already carry non-config state the same way. I added the lifecycle note to the cache doc as suggested.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,411 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_NAME_NODE};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // `Operator::from_config` bypasses the builder's empty-string guard, and
+ // `Some("")` would shadow the path-authority fallback below.
+ if let Some(name_node) = m
+ .remove(HDFS_NAME_NODE)
+ .map(|s| s.trim().trim_end_matches('/').to_string())
Review Comment:
The duplicate-cache part doesn't happen: the configured list is the only key inside a storage, so paths with different authorities still share one operator (test extended to pin that), and opendal strips trailing slashes per entry anyway. But trimming per entry is still the right call, for a different reason: a space after the comma (`a, b`) was passed through as-is and silently broke failover to that NameNode; I verified that against a local listener. Now each entry is trimmed and empties dropped, with tests.
##########
crates/storage/opendal/src/lib.rs:
##########
@@ -135,6 +144,9 @@ pub enum OpenDalStorageFactory {
/// GCS storage factory.
#[cfg(feature = "opendal-gcs")]
Gcs,
+ /// HDFS storage factory.
+ #[cfg(feature = "opendal-hdfs-native")]
+ HdfsNative,
Review Comment:
I'd keep `HdfsNative`. blackmwk's request was actually to rename the module to `hdfs_native` (I renamed the variants to match and offered to revert; nobody asked). opendal has two HDFS services, `services-hdfs` (libhdfs/JNI) and `services-hdfs-native`, and #1130's design names them `Hdfs` and `HdfsNative` respectively, with the maintainers wanting both selectable eventually. Calling this one `Hdfs` would take the libhdfs backend's name. The crate already names variants after opendal services (`Azdls`, `Hf`), so this follows the convention.
##########
crates/storage/opendal/tests/file_io_hdfs_test.rs:
##########
@@ -0,0 +1,315 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Integration tests for HDFS FileIO via OpenDAL `services-hdfs-native`.
+//!
+//! These tests need the HDFS fixture in `dev/docker-compose.yaml` and are
+//! skipped when `ICEBERG_TEST_HDFS_ENDPOINT` is not set. The fixture uses
+//! host networking (Linux, or a Docker runtime that supports it), so it sits
+//! behind a compose profile:
+//!
+//! ```text
+//! COMPOSE_PROFILES=hdfs make docker-up
+//! ICEBERG_TEST_HDFS_ENDPOINT=hdfs://localhost:8020 cargo test -p iceberg-storage-opendal \
+//! --features opendal-hdfs-native --test file_io_hdfs_test
+//! ```
+
+#[cfg(feature = "opendal-hdfs-native")]
+mod tests {
+ use std::sync::Arc;
+
+ use bytes::Bytes;
+ use futures::StreamExt;
+ use iceberg::io::{FileIO, FileIOBuilder, HDFS_NAME_NODE};
+ use iceberg_storage_opendal::{OpenDalResolvingStorageFactory, OpenDalStorageFactory};
+ use iceberg_test_utils::{
+ ENV_HDFS_ENDPOINT, get_hdfs_endpoint, normalize_test_name_with_parts, set_up,
+ };
+
+ /// Skips the calling test unless the HDFS fixture endpoint is configured;
+ /// an unset *or* empty variable means "not provided" (see the HF tests).
+ macro_rules! require_hdfs {
+ () => {
+ match std::env::var(ENV_HDFS_ENDPOINT) {
+ Ok(v) if !v.is_empty() => {}
+ _ => {
+ eprintln!("Skipping HDFS test: {} not set", ENV_HDFS_ENDPOINT);
+ return;
+ }
+ }
+ };
+ }
+
+ fn get_file_io() -> FileIO {
+ set_up();
+ FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative)).build()
+ }
+
+ fn test_path(suffix: &str) -> String {
+ format!(
+ "{}/{}",
+ get_hdfs_endpoint(),
+ normalize_test_name_with_parts!(suffix)
+ )
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_exists() {
+ require_hdfs!();
+ let file_io = get_file_io();
+
+ let absent = test_path("test_file_io_hdfs_exists_absent");
+ assert!(!file_io.exists(&absent).await.unwrap());
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_write_and_read() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_write_and_read");
+ let _ = file_io.delete(&path).await;
+
+ let output = file_io.new_output(&path).unwrap();
+ output
+ .write(Bytes::from_static(b"hello hdfs"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ let input = file_io.new_input(&path).unwrap();
+ assert_eq!(
+ input.read().await.unwrap(),
+ Bytes::from_static(b"hello hdfs")
+ );
+ }
+
+ /// The HA flow: table locations carry a logical authority while
+ /// `hdfs.name-node` carries the (comma-separated) endpoints; it wins.
+ #[tokio::test]
+ async fn test_file_io_hdfs_configured_name_node() {
+ require_hdfs!();
+ set_up();
+ let file_io = FileIOBuilder::new(Arc::new(OpenDalStorageFactory::HdfsNative))
+ .with_prop(HDFS_NAME_NODE, get_hdfs_endpoint())
+ .build();
+
+ // The path authority is a logical name; the configured NameNode wins.
+ let path = format!(
+ "hdfs://logical-nameservice/{}",
+ normalize_test_name_with_parts!("test_file_io_hdfs_configured_name_node")
+ );
+ let _ = file_io.delete(&path).await;
+
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(b"via configured name node"))
+ .await
+ .unwrap();
+
+ assert!(file_io.exists(&path).await.unwrap());
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"via configured name node")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_overwrite() {
+ require_hdfs!();
+ let file_io = get_file_io();
+ let path = test_path("test_file_io_hdfs_overwrite");
+ let _ = file_io.delete(&path).await;
+
+ for content in [b"first".as_slice(), b"second, longer".as_slice()] {
+ file_io
+ .new_output(&path)
+ .unwrap()
+ .write(Bytes::from_static(content))
+ .await
+ .unwrap();
+ }
+
+ assert_eq!(
+ file_io.new_input(&path).unwrap().read().await.unwrap(),
+ Bytes::from_static(b"second, longer")
+ );
+ }
+
+ #[tokio::test]
+ async fn test_file_io_hdfs_delete_stream() {
Review Comment:
Added both. `test_file_io_hdfs_delete_stream_two_name_nodes` writes under `localhost` and `127.0.0.1`, checks each spelling sees the other's file, deletes through one stream and checks all four spellings are gone. Since the fixture is a single cluster it exercises the two-batch path rather than cross-cluster routing, which the batch-key unit tests pin. `test_hdfs_native_operator_cache_keeps_first_insert` inserts two distinguishable operators for one key, asserts the first is kept and returned to both callers, then races eight threads through `create_operator`.
##########
crates/test_utils/src/lib.rs:
##########
@@ -47,13 +47,15 @@ mod common {
pub const ENV_HMS_ENDPOINT: &str = "ICEBERG_TEST_HMS_ENDPOINT";
pub const ENV_GLUE_ENDPOINT: &str = "ICEBERG_TEST_GLUE_ENDPOINT";
pub const ENV_GCS_ENDPOINT: &str = "ICEBERG_TEST_GCS_ENDPOINT";
+ pub const ENV_HDFS_ENDPOINT: &str = "ICEBERG_TEST_HDFS_ENDPOINT";
// Default ports matching dev/docker-compose.yaml
pub const DEFAULT_OBJECT_STORE_PORT: u16 = 9000;
pub const DEFAULT_REST_CATALOG_PORT: u16 = 8181;
pub const DEFAULT_HMS_PORT: u16 = 9083;
pub const DEFAULT_GLUE_PORT: u16 = 5001;
pub const DEFAULT_GCS_PORT: u16 = 4443;
+ pub const DEFAULT_HDFS_NN_PORT: u16 = 8020;
Review Comment:
Renamed to `DEFAULT_HDFS_PORT`.
##########
crates/iceberg/src/io/storage/config/hdfs.rs:
##########
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage configuration.
+
+/// HDFS NameNode RPC endpoint(s), e.g. `hdfs://namenode:8020`; a
+/// comma-separated list enables HA failover. When unset, the NameNode is
+/// derived from the path authority.
+pub const HDFS_NAME_NODE: &str = "hdfs.name-node";
+/// Prefix for properties forwarded to the HDFS client configuration, e.g.
+/// `hadoop.dfs.client.failover.random.order`. Forwarded values (prefix
+/// stripped) override those loaded from `$HADOOP_CONF_DIR`.
+pub const HDFS_HADOOP_CONF_PREFIX: &str = "hadoop.";
Review Comment:
Reworded: the doc now says `hadoop.dfs.client.failover.random.order` is forwarded as `dfs.client.failover.random.order`, and mentions `hadoop.fs.defaultFS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4193471358
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
+ .or_else(|| hdfs_native_default_fs(config))
+ .ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid hdfs path: {path}, authority-less paths require `{HDFS_NAME_NODE}` or `{HDFS_HOST}`"
+ ),
+ )
+ })?;
+ Ok((name_node, relative_path))
+}
+
+/// `fs.defaultFS` from the forwarded options, when it is an HDFS URI.
+fn hdfs_native_default_fs(config: &HdfsNativeConfig) -> Option<String> {
+ config
+ .options
+ .as_ref()?
+ .get(FS_DEFAULT_FS)
+ .map(|s| s.trim().trim_end_matches('/'))
+ .filter(|s| {
+ s.strip_prefix("hdfs://")
+ .is_some_and(|rest| !rest.is_empty())
+ })
+ .map(str::to_string)
+}
+
+/// Operators cached per effective NameNode: each holds an `hdfs-native`
+/// client with live RPC connections, whose tasks run on the tokio runtime
+/// current when it was built (a private one when built outside any). The
+/// cache lives as long as the storage that owns it (clones share it) and
+/// never evicts, so the storage must not be used from another runtime once
+/// the building one is dropped: `hdfs-native` panics on a dead runtime.
+#[derive(Clone, Debug, Default)]
+pub struct HdfsNativeOperatorCache(Arc<RwLock<HashMap<String, Operator>>>);
Review Comment:
Fixed rather than documented. tokio's runtime id is unstable, so the cache tracks liveness instead: each entry parks a tiny sentinel task on the runtime that built it and keeps a `Weak` to a token the task owns; once that runtime is gone the token dies, the entry reads as stale and the operator is rebuilt on the current runtime. Sentinels are aborted when entries drop, so nothing accumulates. Covered by a unit test for both runtime flavors, a task-count test, and `tests/hdfs_native_runtime_test.rs`, which reuses a `FileIO` from a second runtime after the first is dropped; that test panics on the old cache and passes now. Public API is unchanged. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6017029091
Thanks @laskoviymishka. All 10 threads are addressed in 22db8e6..f83db4e (merge of main plus 8 commits), each with an inline reply. Behavior changes: the operator cache rebuilds entries whose tokio runtime is gone instead of hdfs-native panicking (regression test included); NameNode resolution follows Hadoop, so a `host:port` authority is never overridden and portless names must resolve through `hdfs.name-node`; the HDFS variant wraps one opaque `HdfsNativeStorage`; operator builds run under `spawn_blocking`; the HDFS integration tests moved to a dedicated CI job; ignored or malformed properties now warn or error instead of being dropped. Description updated; full fixture suite green locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4193890822
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
+ entry
+ .rsplit_once(':')
+ .is_none_or(|(_, port)| port.parse::<u16>().is_err())
+ }) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_NAME_NODE}` entry: {entry}, expected host:port"),
+ ));
+ }
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ let host = m.remove(HDFS_HOST).map(|s| s.trim().to_string());
+ let port = m.remove(HDFS_PORT).map(|s| s.trim().to_string());
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ if let Some(host) = host.filter(|s| !s.is_empty()) {
+ let port = match port.filter(|s| !s.is_empty()) {
+ Some(port) => port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })?,
+ None => HDFS_DEFAULT_PORT,
+ };
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path — the configured
+/// `hdfs.name-node` when set, else the path authority, else `fs.defaultFS` —
+/// plus the relative path. The operator cache, `delete_stream` batching and
+/// `relativize_path` all go through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority_name_node, relative_path) = hdfs_native_parse_path(path)?;
+ let name_node = config
+ .name_node
+ .clone()
+ .or(authority_name_node)
Review Comment:
Agreed, and Hadoop semantics settle it: an authority with a port is now used as is, and `hdfs.name-node` applies only to logical nameservice authorities (no port) and authority-less paths. Verified against local listeners that `hdfs://other:9000/...` dials `other:9000` even with the property set, and pinned by `test_hdfs_native_create_operator_configured_name_node_serves_logical_paths`, which puts a logical, an authority-less and a concrete path through one storage and gets two operators. Doc updated. Lands with the next push.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,635 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::sync::{Arc, RwLock};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use url::Url;
+
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries: Vec<&str> = name_node
+ .split(',')
+ .map(|entry| entry.trim().trim_end_matches('/'))
+ .filter(|entry| !entry.is_empty())
+ .collect();
+ // hdfs-native dials each entry as `host:port` (`hdfs://` optional);
+ // without a port it fails only at the first I/O.
+ if let Some(entry) = entries.iter().find(|entry| {
Review Comment:
Confirmed: hdfs-native dials each NameNode as a socket address and has no default port, so a portless string fails at first I/O with "invalid socket address" from any of the three sources. One rule now: `host:port` is dialed; anything portless is a logical nameservice that only `hdfs.name-node` resolves. A portless authority or `fs.defaultFS` without the property errors at resolution, naming the nameservice and the property; `hdfs.name-node` entries keep the build-time check. Covered by the precedence test and the `fs.defaultFS` test. Lands with the next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#discussion_r4198203938
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
+ url.port()
+ .map(|port| format!("hdfs://{host}:{port}"))
+ .unwrap_or_else(|| format!("hdfs://{host}"))
+ });
+
+ // `url.path()` borrows from `url` and can't be returned with the input's
+ // lifetime. Slice the path component out of the original input instead;
+ // it starts after the first `/` following the `hdfs://` prefix. Opendal
+ // paths must not start with `/` (`Deleter::delete` rejects them).
+ let rel = match after_scheme.find('/') {
+ Some(i) => after_scheme[i..].trim_start_matches('/'),
+ None => "",
+ };
+
+ Ok((name_node, rel))
+}
+
+/// Resolves the effective NameNode for a path, plus the relative path. As in
+/// Hadoop, an authority with a port is used as is; a logical nameservice
+/// authority (no port) resolves through `hdfs.name-node`, and an
+/// authority-less path through `hdfs.name-node`, else `fs.defaultFS`. The
+/// operator cache, `delete_stream` batching and `relativize_path` all go
+/// through this, so they cannot drift apart.
+pub(crate) fn hdfs_native_effective_name_node<'a>(
+ config: &HdfsNativeConfig,
+ path: &'a str,
+) -> Result<(String, &'a str)> {
+ let (authority, relative_path) = hdfs_native_parse_path(path)?;
+ let invalid = |reason: String| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, {reason}"),
+ )
+ };
+ let name_node = match authority {
+ Some(authority) if hdfs_native_name_node(&authority).is_some() => authority,
+ Some(logical) => {
+ let nameservice = logical.trim_start_matches("hdfs://");
+ match hdfs_native_nameservice(config, nameservice)? {
+ Some(name_node) => name_node,
+ None => config.name_node.clone().ok_or_else(|| {
Review Comment:
With the plain `hdfs.name-node` set, any portless authority that isn't a declared nameservice lands here and resolves to that one cluster — so `hdfs://ns-typo/...` or a bare second-cluster hostname gets read, written, and `delete_prefix`'d against the wrong NameNode, no error. On a single cluster the fallback is accidentally harmless; on anything multi-cluster it's a silent wrong-cluster delete. I'd require the declaration (`hdfs.name-node.<ns>`) for logical authorities and keep the plain key for authority-less paths only — or, if the single-cluster convenience matters, fall back only when the authority matches the `fs.defaultFS` nameservice. Either way an unknown name should error rather than guess.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
Review Comment:
small one, not blocking — `hdfs.host`/`hdfs.port` aren't validated before composing, so `hdfs.host=nn:8020` becomes `hdfs://[nn:8020]:8020` and `hdfs.port=0` becomes `hdfs://host:0`, both failing later with a confusing error. Running the composed value through `hdfs_native_name_node` here and erroring on `hdfs.host`/`hdfs.port` would catch it at config time.
##########
crates/storage/opendal/src/hdfs_native.rs:
##########
@@ -0,0 +1,1244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HDFS storage backend via OpenDAL's `services-hdfs-native` (pure Rust, no JNI).
+
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::sync::{Arc, RwLock, Weak};
+
+use iceberg::io::{HDFS_HADOOP_CONF_PREFIX, HDFS_HOST, HDFS_NAME_NODE, HDFS_PORT};
+use iceberg::{Error, ErrorKind, Result};
+use opendal::Operator;
+use opendal::services::HdfsNativeConfig;
+use serde::{Deserialize, Serialize};
+use tokio::runtime::Handle;
+use tokio::task::JoinHandle;
+use url::Url;
+
+use crate::OpenDalClientConfig;
+use crate::utils::from_opendal_error;
+
+/// Hadoop's default filesystem, which serves authority-less paths.
+const FS_DEFAULT_FS: &str = "fs.defaultFS";
+const HDFS_DEFAULT_PORT: u16 = 8020;
+/// PyIceberg keys with no equivalent in opendal's config.
+const HDFS_UNSUPPORTED_KEYS: [&str; 2] = ["hdfs.user", "hdfs.kerberos_ticket"];
+/// Hadoop's keys declaring an HA nameservice: `hdfs.name-node.<nameservice>`
+/// expands to them, and they are honored as well when passed through `hadoop.`.
+const HA_NAMENODES_PREFIX: &str = "dfs.ha.namenodes";
+const HA_NAMENODE_RPC_ADDRESS_PREFIX: &str = "dfs.namenode.rpc-address";
+
+/// Normalizes one NameNode spelling to `hdfs://host:port`, the form path
+/// authorities take, so every source shares cache keys. `hdfs-native` dials
+/// a socket address and has no default port, so anything else is `None`:
+/// another scheme, a logical name, port 0, userinfo, a path.
+fn hdfs_native_name_node(entry: &str) -> Option<String> {
+ let rest = entry.trim().trim_end_matches('/');
+ let rest = rest.strip_prefix("hdfs://").unwrap_or(rest);
+ if rest.is_empty() || rest.contains("://") {
+ return None;
+ }
+ let url = Url::parse(&format!("hdfs://{rest}")).ok()?;
+ let (host, port) = (url.host_str()?, url.port()?);
+ let plain = host.is_empty()
+ || port == 0
+ || !url.username().is_empty()
+ || url.password().is_some()
+ || !matches!(url.path(), "" | "/")
+ || url.query().is_some()
+ || url.fragment().is_some();
+ (!plain).then(|| format!("hdfs://{host}:{port}"))
+}
+
+/// Parses a comma-separated NameNode list, each entry normalized; an empty
+/// list is `Ok` and empty.
+fn hdfs_native_name_node_list(property: &str, value: &str) -> Result<Vec<String>> {
+ value
+ .split(',')
+ .map(str::trim)
+ .filter(|entry| !entry.is_empty())
+ .map(|entry| {
+ hdfs_native_name_node(entry).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid `{property}` entry: {entry}, expected host:port (hdfs:// optional)"
+ ),
+ )
+ })
+ })
+ .collect()
+}
+
+/// Parse iceberg properties to [`HdfsNativeConfig`].
+pub(crate) fn hdfs_native_config_parse(mut m: HashMap<String, String>) -> Result<HdfsNativeConfig> {
+ let mut cfg = HdfsNativeConfig::default();
+
+ // Entries are trimmed one by one: opendal splits the list on `,` as is,
+ // so a space after a comma would break failover to that NameNode. An
+ // empty result is dropped because `Operator::from_config` bypasses the
+ // builder's empty-string guard and `Some("")` would shadow the
+ // path-authority fallback below.
+ if let Some(name_node) = m.remove(HDFS_NAME_NODE) {
+ let entries = hdfs_native_name_node_list(HDFS_NAME_NODE, &name_node)?;
+ if !entries.is_empty() {
+ cfg.name_node = Some(entries.join(","));
+ }
+ }
+
+ // `hdfs.name-node.<nameservice>` is sugar for Hadoop's own declaration of
+ // an HA nameservice, which the resolver reads back from the options.
+ let nameservice_prefix = format!("{HDFS_NAME_NODE}.");
+ let declared_keys: Vec<String> = m
+ .keys()
+ .filter(|key| key.starts_with(&nameservice_prefix))
+ .cloned()
+ .collect();
+ let mut declared = Vec::new();
+ for key in declared_keys {
+ let value = m.remove(&key).unwrap_or_default();
+ let nameservice = key[nameservice_prefix.len()..].to_string();
+ if nameservice.is_empty() || nameservice.chars().any(char::is_whitespace) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{key}`: a nameservice name must follow `{nameservice_prefix}`"
+ ),
+ ));
+ }
+ let entries = hdfs_native_name_node_list(&key, &value)?;
+ if entries.is_empty() {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid property `{key}`: no NameNodes"),
+ ));
+ }
+ declared.push((nameservice, entries));
+ }
+
+ // A config carried over from PyIceberg would otherwise change identity
+ // silently; the client reads `HADOOP_USER_NAME` and the Kerberos cache.
+ for key in HDFS_UNSUPPORTED_KEYS {
+ if m.remove(key).is_some() {
+ tracing::warn!("`{key}` is not supported by the hdfs-native backend and is ignored");
+ }
+ }
+ if m.contains_key(HDFS_HADOOP_CONF_PREFIX) {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Invalid property `{HDFS_HADOOP_CONF_PREFIX}`: a Hadoop key must follow the prefix"
+ ),
+ ));
+ }
+
+ let host = m
+ .remove(HDFS_HOST)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty());
+ let port = m
+ .remove(HDFS_PORT)
+ .map(|s| s.trim().to_string())
+ .filter(|s| !s.is_empty())
+ .map(|port| {
+ port.parse::<u16>().map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid `{HDFS_PORT}`: {port}: {e}"),
+ )
+ })
+ })
+ .transpose()?;
+
+ let mut options: HashMap<String, String> = m
+ .into_iter()
+ .filter_map(|(key, value)| {
+ key.strip_prefix(HDFS_HADOOP_CONF_PREFIX)
+ .map(|stripped| (stripped.to_string(), value))
+ })
+ .collect();
+ // Explicit `hadoop.` keys win over the sugar.
+ for (nameservice, entries) in declared {
+ let ids: Vec<String> = (0..entries.len()).map(|i| format!("nn{i}")).collect();
+ options
+ .entry(format!("{HA_NAMENODES_PREFIX}.{nameservice}"))
+ .or_insert_with(|| ids.join(","));
+ for (id, entry) in ids.iter().zip(&entries) {
+ options
+ .entry(format!(
+ "{HA_NAMENODE_RPC_ADDRESS_PREFIX}.{nameservice}.{id}"
+ ))
+ .or_insert_with(|| entry.trim_start_matches("hdfs://").to_string());
+ }
+ }
+ // PyIceberg's `hdfs.host`/`hdfs.port` name the filesystem for
+ // authority-less paths, which is what Hadoop's `fs.defaultFS` means; an
+ // explicit `hadoop.fs.defaultFS` wins.
+ match host {
+ Some(host) => {
+ // An IPv6 literal needs brackets in a URI authority.
+ let host = if host.contains(':') && !host.starts_with('[') {
+ format!("[{host}]")
+ } else {
+ host
+ };
+ let port = port.unwrap_or(HDFS_DEFAULT_PORT);
+ options
+ .entry(FS_DEFAULT_FS.to_string())
+ .or_insert_with(|| format!("hdfs://{host}:{port}"));
+ }
+ None if port.is_some() => {
+ tracing::warn!("`{HDFS_PORT}` has no effect without `{HDFS_HOST}` and is ignored");
+ }
+ None => {}
+ }
+ if !options.is_empty() {
+ cfg.options = Some(options);
+ }
+
+ Ok(cfg)
+}
+
+/// Parse an HDFS path into `Some("hdfs://<authority>")` (`None` when
+/// authority-less) and the relative path (no leading `/`, opendal style).
+pub(crate) fn hdfs_native_parse_path(path: &str) -> Result<(Option<String>, &str)> {
+ let url = Url::parse(path).map_err(|e| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}: {e}"),
+ )
+ })?;
+ // Non-special schemes parse even without `//` (e.g. `hdfs:x` is a valid
+ // non-hierarchical URL), so require the literal prefix before slicing.
+ let (Some(after_scheme), "hdfs") = (path.strip_prefix("hdfs://"), url.scheme()) else {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid hdfs path: {path}, expected scheme `hdfs://`"),
+ ));
+ };
+
+ let name_node = url.host_str().filter(|h| !h.is_empty()).map(|host| {
Review Comment:
Two gaps here that the config side already closes: userinfo is silently dropped (`hdfs://alice@nn:8020/a` resolves as `nn:8020`, while `hdfs_native_name_node` rejects userinfo for configured values), and port 0 passes through as `hdfs://nn:0` — which then fails `hdfs_native_name_node`, gets re-read as a logical nameservice `nn:0`, and takes the fallback below. I'd reject both here with `DataInvalid`; an authority that carries a port but fails validation should be an error, not a nameservice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(storage): Add HDFS support via opendal services-hdfs-native [iceberg-rust]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #3111:
URL: https://github.com/apache/iceberg-rust/pull/3111#issuecomment-6345651713
Stupendous 🔥
Cannot wait to use it! May have to speed forward on the iceberg-rust release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org