You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lrsb (via GitHub)" <gi...@apache.org> on 2026/04/17 15:54:38 UTC
[PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
lrsb opened a new pull request, #16018:
URL: https://github.com/apache/iceberg/pull/16018
Fixes https://github.com/apache/iceberg/issues/16017
HadoopFileIO accepts catalog properties through FileIO.initialize, but there is currently no way to push Hadoop-level settings (fs.s3a.*, fs.defaultFS, etc.) through the catalog properties. Callers who want to parameterize those values per catalog end up having to mutate the shared Hadoop Configuration out-of-band, which is awkward and leaks settings between catalogs that share a Configuration instance.
Copy the base Configuration and overlay any property whose key starts with "iceberg.hadoop.conf." (prefix stripped). Both initialize and setConf delegate to the same helper, so the feature works for both the REST-catalog-driven and Hadoop-aware (`HadoopConfigurable#setConf`) paths. The original base Configuration is never mutated.
Unrelated catalog properties are not copied to the Hadoop Configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16018:
URL: https://github.com/apache/iceberg/pull/16018#issuecomment-4760474471
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16018: Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration
URL: https://github.com/apache/iceberg/pull/16018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16018:
URL: https://github.com/apache/iceberg/pull/16018#issuecomment-4828073535
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16018:
URL: https://github.com/apache/iceberg/pull/16018#issuecomment-4473193205
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Propagate iceberg.hadoop.conf. properties to Hadoop Configuration [iceberg]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #16018:
URL: https://github.com/apache/iceberg/pull/16018#issuecomment-4509648848
Hi @lrsb , thanks for your first PR to Iceberg!
I have some concerns about the PR though.
The prefix-strip, Hadoop conf mechanism has a clear precedent in
Spark's `spark.sql.catalog.<name>.hadoop.*` (SparkUtil.hadoopConfCatalogOverrides),
which is also an unbounded receiver it sets whatever key/value pairs match the prefix without filtering.
However to your credit: There is a structural difference on the source side.
Spark's mechanism reads from SQLConf, populated only from local sources
(spark-defaults.conf, CLI flags, programmatic config, SQL SET). No remote
party can push into it, making it harder to maintain a consistent coordinated.
This PR reads from `FileIO.initialize(properties)`, which in the
`RESTSessionCatalog` path is the result of `ConfigResponse.merge()`,
so it can include the server's `/v1/config` `overrides` map.
**However, introducing this feature is a meaningful design choice
that probably deserves explicit discussion on the dev list before
locking it in as the de facto remote convention for
catalog-property to Hadoop-conf propagation:**
1. It becomes a default convention once shipped: a REST catalog that
starts pushing `iceberg.hadoop.conf.fs.s3a.endpoint=...` is
compatible with any client that uses this feature, but
reverting the feature later is hard.
2. Closing it down (e.g. via allowlist) after the fact is a
compat break; opening it up later is not.
Three options worth discussing, ordered from most-permissive to least:
a) Ship as-is, but document the open-ended nature in the Javadoc
on the new constant and in the docs page. (Not recommended from my side)
b) Mirror S3FileIOProperties: curate an explicit list of safe
keys (`fs.s3a.endpoint`, `fs.azure.account.key.*`, etc.) as
named constants, accept those, ignore others.
c) Keep the prefix open-ended but make it opt-in via a client-set
flag (e.g. `iceberg.hadoop.conf.enabled=true`).
I lean toward (c) for shipping if you have a real use case plus a follow-up to discuss (b).
**But before any of that I think this PR really needs a design discussion on the dev list.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org