You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "youngrae81 (via GitHub)" <gi...@apache.org> on 2026/10/27 06:34:13 UTC
[PR] Add a documentation page for Hadoop catalog federation [polaris]
youngrae81 opened a new pull request, #5883:
URL: https://github.com/apache/polaris/pull/5883
## Summary
Adds `site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md`,
next to the Hive and BigQuery pages, and moves the BigQuery page one weight down so the section reads REST, Hive, Hadoop, BigQuery.
Hadoop federation is wired into the default server build, but the only place it was described was
the CLI reference. The new page follows the structure of the Hive page:
- Build-time enablement: none needed, the extension is always included.
- Feature flags: `SUPPORTED_CATALOG_CONNECTION_TYPES`, `SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES`, `ENABLE_CATALOG_FEDERATION`.
- Runtime requirements: the server reads and writes the warehouse itself, Hadoop configuration discovery, `IMPLICIT` authentication only.
- Warehouse access patterns, including that an HDFS warehouse cannot be described in `storageConfigInfo` today: `hdfs://` is not an accepted location prefix and `default-base-location` must lie within an allowed location.
- Creating the catalog with the Management API and the CLI, and the `uri` restriction: it is validated as a `java.net.URL`, so `hdfs://` and `s3://` are rejected.
- Connection properties forwarded since #5680 (`suppress-permission-error`, `s3.endpoint`, `s3.path-style-access`, `client.region`).
- Limitations: single identity, no generic tables, directory-layout discovery, no namespace properties.
All statements were checked against `HadoopFederatedCatalogFactory`, `HadoopConnectionConfigInfoDpo`, `ConnectionConfigInfoDpo#validateUri`, `FileStorageConfigurationInfo`, `StorageLocationValidator` and Iceberg's `HadoopCatalog`. The site builds with `site/bin/create-static-site.sh`.
Fixes #5865
## Checklist
- [x] I have read the [contributing guidelines](https://github.com/apache/polaris/blob/main/CONTRIBUTING.md)
- [x] This PR has a clear title and description
- [ ] I have added tests for the change (documentation only)
- [x] All tests pass locally (site build)
- [ ] I have updated the CHANGELOG (documentation only, no behavior change)
- [x] I have updated the documentation
AI assistance was used to draft the page; every claim was verified against the code before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain merged PR #5883:
URL: https://github.com/apache/polaris/pull/5883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4403539865
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
Review Comment:
We don't have to mention this given that Polaris' responsibility here is federation.
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher
+ scripts, which the Polaris server and image do not use, so place those files on the Polaris
+ server classpath, for example through an image layer, when the warehouse is on HDFS or needs
+ non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. Table loads from a federated catalog also check every table location against `allowedLocations` before vending credentials (unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`), so an `hdfs://` table location fails at load time too. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts these location checks but still needs a `file://` base location next to it. |
+| `HadoopFileIO` with `s3a://` warehouses | Only with extra runtime packaging | `hadoop-aws` and related file system dependencies are not included in the Polaris runtime. |
+| `S3FileIO` with ambient AWS credentials | Sometimes | Set `io-impl=org.apache.iceberg.aws.s3.S3FileIO` in the catalog-level `properties`. Commits still rely on atomic rename, which S3 does not provide, so concurrent writers are unsafe. |
+| **UNSAFE!** Putting object-store credentials in catalog properties | **DO NOT USE / THIS IS UNSAFE** | Secrets placed in catalog properties are returned to authenticated catalog clients through `/config`. This **exposes passwords, access keys, session tokens, or other secrets as plain text**. |
+
+## Creating a federated catalog
+
+Use the Management API (or the Python CLI) to create an external catalog whose connection type is
+`HADOOP`. The following request registers a catalog whose tables live under
+`file:///data/warehouse/analytics`, a directory mounted into the Polaris container:
+
+```bash
+curl -X POST https://<polaris-host>/management/v1/catalogs \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "type": "EXTERNAL",
+ "name": "analytics_hadoop",
+ "storageConfigInfo": {
+ "storageType": "FILE",
+ "allowedLocations": ["file:///data/warehouse/analytics/"]
+ },
+ "properties": { "default-base-location": "file:///data/warehouse/analytics/" },
+ "connectionConfigInfo": {
+ "connectionType": "HADOOP",
+ "uri": "file:///data/warehouse/analytics/",
+ "warehouse": "file:///data/warehouse/analytics/",
+ "authenticationParameters": { "authenticationType": "IMPLICIT" }
+ }
+ }'
+```
+
+The same catalog can be created with the CLI:
+
+```bash
+polaris catalogs create analytics_hadoop \
+ --type external \
+ --storage-type file \
Review Comment:
Not a blocker: I'd suggest a s3 or hdfs type, so that this doc is closer to real use cases.
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher
+ scripts, which the Polaris server and image do not use, so place those files on the Polaris
+ server classpath, for example through an image layer, when the warehouse is on HDFS or needs
+ non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. Table loads from a federated catalog also check every table location against `allowedLocations` before vending credentials (unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`), so an `hdfs://` table location fails at load time too. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts these location checks but still needs a `file://` base location next to it. |
+| `HadoopFileIO` with `s3a://` warehouses | Only with extra runtime packaging | `hadoop-aws` and related file system dependencies are not included in the Polaris runtime. |
+| `S3FileIO` with ambient AWS credentials | Sometimes | Set `io-impl=org.apache.iceberg.aws.s3.S3FileIO` in the catalog-level `properties`. Commits still rely on atomic rename, which S3 does not provide, so concurrent writers are unsafe. |
+| **UNSAFE!** Putting object-store credentials in catalog properties | **DO NOT USE / THIS IS UNSAFE** | Secrets placed in catalog properties are returned to authenticated catalog clients through `/config`. This **exposes passwords, access keys, session tokens, or other secrets as plain text**. |
+
+## Creating a federated catalog
+
+Use the Management API (or the Python CLI) to create an external catalog whose connection type is
+`HADOOP`. The following request registers a catalog whose tables live under
+`file:///data/warehouse/analytics`, a directory mounted into the Polaris container:
+
+```bash
+curl -X POST https://<polaris-host>/management/v1/catalogs \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "type": "EXTERNAL",
+ "name": "analytics_hadoop",
+ "storageConfigInfo": {
+ "storageType": "FILE",
+ "allowedLocations": ["file:///data/warehouse/analytics/"]
+ },
+ "properties": { "default-base-location": "file:///data/warehouse/analytics/" },
+ "connectionConfigInfo": {
+ "connectionType": "HADOOP",
+ "uri": "file:///data/warehouse/analytics/",
+ "warehouse": "file:///data/warehouse/analytics/",
+ "authenticationParameters": { "authenticationType": "IMPLICIT" }
+ }
+ }'
+```
+
+The same catalog can be created with the CLI:
+
+```bash
+polaris catalogs create analytics_hadoop \
+ --type external \
+ --storage-type file \
+ --default-base-location file:///data/warehouse/analytics/ \
+ --catalog-connection-type hadoop \
+ --catalog-uri file:///data/warehouse/analytics/ \
+ --hadoop-warehouse file:///data/warehouse/analytics/ \
+ --catalog-authentication-type implicit
+```
+
+`warehouse` is the root directory that the `HadoopCatalog` scans for namespaces and tables. `uri`
+is required by the connection configuration API but is not used by the Hadoop catalog. Polaris
+validates it as a `java.net.URL`, so it must use a scheme the JVM knows (`file`, `http`, `https`);
+`hdfs://` and `s3://` values are rejected with `Invalid remote URI`. For a `file://` warehouse,
Review Comment:
This claim seems valid. If `hdfs://` and `s3://` values are rejected in the `uri`, also I checked a bit `uri` seems not needed for the connection. Maybe we should ignore it in the code, and document it clearly here.
cc @XJDKC @dennishuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4397547169
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,191 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. The Java client does not consult `HADOOP_CONF_DIR` by itself, so place
+ those files on the Polaris server classpath, for example through an image layer, when the
+ warehouse is on HDFS or needs non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts the runtime location checks but still needs a `file://` base location next to it. |
Review Comment:
Thanks, confirmed. `buildLoadTableResponseWithDelegationCredentials` runs on every table load, and for a federated catalog it calls `validateTableLocations` unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`, so an `hdfs://` table location is rejected at load time with a 403 even if the catalog itself could be created. I did not run an HDFS cluster end to end; the conclusion follows from the storage type prefix rules and that check.
The latest commit adds that load-time check to this row and changes the Hive row to match ("Only with a `*` wildcard in `allowedLocations`"), with a link back to this page, so the two tables agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4407230438
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher
+ scripts, which the Polaris server and image do not use, so place those files on the Polaris
+ server classpath, for example through an image layer, when the warehouse is on HDFS or needs
+ non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. Table loads from a federated catalog also check every table location against `allowedLocations` before vending credentials (unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`), so an `hdfs://` table location fails at load time too. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts these location checks but still needs a `file://` base location next to it. |
+| `HadoopFileIO` with `s3a://` warehouses | Only with extra runtime packaging | `hadoop-aws` and related file system dependencies are not included in the Polaris runtime. |
+| `S3FileIO` with ambient AWS credentials | Sometimes | Set `io-impl=org.apache.iceberg.aws.s3.S3FileIO` in the catalog-level `properties`. Commits still rely on atomic rename, which S3 does not provide, so concurrent writers are unsafe. |
+| **UNSAFE!** Putting object-store credentials in catalog properties | **DO NOT USE / THIS IS UNSAFE** | Secrets placed in catalog properties are returned to authenticated catalog clients through `/config`. This **exposes passwords, access keys, session tokens, or other secrets as plain text**. |
+
+## Creating a federated catalog
+
+Use the Management API (or the Python CLI) to create an external catalog whose connection type is
+`HADOOP`. The following request registers a catalog whose tables live under
+`file:///data/warehouse/analytics`, a directory mounted into the Polaris container:
+
+```bash
+curl -X POST https://<polaris-host>/management/v1/catalogs \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "type": "EXTERNAL",
+ "name": "analytics_hadoop",
+ "storageConfigInfo": {
+ "storageType": "FILE",
+ "allowedLocations": ["file:///data/warehouse/analytics/"]
+ },
+ "properties": { "default-base-location": "file:///data/warehouse/analytics/" },
+ "connectionConfigInfo": {
+ "connectionType": "HADOOP",
+ "uri": "file:///data/warehouse/analytics/",
+ "warehouse": "file:///data/warehouse/analytics/",
+ "authenticationParameters": { "authenticationType": "IMPLICIT" }
+ }
+ }'
+```
+
+The same catalog can be created with the CLI:
+
+```bash
+polaris catalogs create analytics_hadoop \
+ --type external \
+ --storage-type file \
+ --default-base-location file:///data/warehouse/analytics/ \
+ --catalog-connection-type hadoop \
+ --catalog-uri file:///data/warehouse/analytics/ \
+ --hadoop-warehouse file:///data/warehouse/analytics/ \
+ --catalog-authentication-type implicit
+```
+
+`warehouse` is the root directory that the `HadoopCatalog` scans for namespaces and tables. `uri`
+is required by the connection configuration API but is not used by the Hadoop catalog. Polaris
+validates it as a `java.net.URL`, so it must use a scheme the JVM knows (`file`, `http`, `https`);
+`hdfs://` and `s3://` values are rejected with `Invalid remote URI`. For a `file://` warehouse,
Review Comment:
Can we file an issue for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "orca-review-bot (via GitHub)" <gi...@apache.org>.
orca-review-bot commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4391073463
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,191 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. The Java client does not consult `HADOOP_CONF_DIR` by itself, so place
+ those files on the Polaris server classpath, for example through an image layer, when the
+ warehouse is on HDFS or needs non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts the runtime location checks but still needs a `file://` base location next to it. |
Review Comment:
This row says HDFS isn't workable, but the Hive page's matching table marks "HDFS via ambient Hadoop config" as Yes/recommended. Both pages follow the same storageConfigInfo prefix rules. I think this page is the correct one. ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING defaults to true, so federated loads run validateTableLocations against allowedLocations (IcebergCatalogHandler ~L1222), and an hdfs:// table location would fail that. Did you try an hdfs:// load end to end? If it does fail, can we fix the Hive row as well, here or in a follow-up, so the two pages agree?
<sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,191 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. The Java client does not consult `HADOOP_CONF_DIR` by itself, so place
Review Comment:
nit: the hadoop launcher scripts do add HADOOP_CONF_DIR to the classpath, so this reads a bit absolute. Can we say it applies to the Polaris server/image?
<sub>🤖 AI-generated review comment — agents make mistakes, so please verify before acting on it.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4407373415
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher
+ scripts, which the Polaris server and image do not use, so place those files on the Polaris
+ server classpath, for example through an image layer, when the warehouse is on HDFS or needs
+ non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. Table loads from a federated catalog also check every table location against `allowedLocations` before vending credentials (unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`), so an `hdfs://` table location fails at load time too. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts these location checks but still needs a `file://` base location next to it. |
+| `HadoopFileIO` with `s3a://` warehouses | Only with extra runtime packaging | `hadoop-aws` and related file system dependencies are not included in the Polaris runtime. |
+| `S3FileIO` with ambient AWS credentials | Sometimes | Set `io-impl=org.apache.iceberg.aws.s3.S3FileIO` in the catalog-level `properties`. Commits still rely on atomic rename, which S3 does not provide, so concurrent writers are unsafe. |
+| **UNSAFE!** Putting object-store credentials in catalog properties | **DO NOT USE / THIS IS UNSAFE** | Secrets placed in catalog properties are returned to authenticated catalog clients through `/config`. This **exposes passwords, access keys, session tokens, or other secrets as plain text**. |
+
+## Creating a federated catalog
+
+Use the Management API (or the Python CLI) to create an external catalog whose connection type is
+`HADOOP`. The following request registers a catalog whose tables live under
+`file:///data/warehouse/analytics`, a directory mounted into the Polaris container:
+
+```bash
+curl -X POST https://<polaris-host>/management/v1/catalogs \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "type": "EXTERNAL",
+ "name": "analytics_hadoop",
+ "storageConfigInfo": {
+ "storageType": "FILE",
+ "allowedLocations": ["file:///data/warehouse/analytics/"]
+ },
+ "properties": { "default-base-location": "file:///data/warehouse/analytics/" },
+ "connectionConfigInfo": {
+ "connectionType": "HADOOP",
+ "uri": "file:///data/warehouse/analytics/",
+ "warehouse": "file:///data/warehouse/analytics/",
+ "authenticationParameters": { "authenticationType": "IMPLICIT" }
+ }
+ }'
+```
+
+The same catalog can be created with the CLI:
+
+```bash
+polaris catalogs create analytics_hadoop \
+ --type external \
+ --storage-type file \
+ --default-base-location file:///data/warehouse/analytics/ \
+ --catalog-connection-type hadoop \
+ --catalog-uri file:///data/warehouse/analytics/ \
+ --hadoop-warehouse file:///data/warehouse/analytics/ \
+ --catalog-authentication-type implicit
+```
+
+`warehouse` is the root directory that the `HadoopCatalog` scans for namespaces and tables. `uri`
+is required by the connection configuration API but is not used by the Hadoop catalog. Polaris
+validates it as a `java.net.URL`, so it must use a scheme the JVM knows (`file`, `http`, `https`);
+`hdfs://` and `s3://` values are rejected with `Invalid remote URI`. For a `file://` warehouse,
Review Comment:
Agreed.
`HadoopFederatedCatalogFactory` only reads `warehouse`; `uri` never reaches the `HadoopCatalog`.
The paragraph now says up front that the value is ignored and only explains the current URL validation as the reason a placeholder is needed. Filed #5905 to relax or drop the `uri` requirement for `HADOOP` connections so the code and the doc can be aligned there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4407367126
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
Review Comment:
Agreed, removed in the latest commit. The S3 row in the access-pattern table still mentions the concurrent-writer caveat where it is directly relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4397551611
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,191 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. The Java client does not consult `HADOOP_CONF_DIR` by itself, so place
Review Comment:
Good point.
Reworded: `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher scripts, which the Polaris server and image do not use, so the files need to be on the server classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4397547169
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,191 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. The Java client does not consult `HADOOP_CONF_DIR` by itself, so place
+ those files on the Polaris server classpath, for example through an image layer, when the
+ warehouse is on HDFS or needs non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts the runtime location checks but still needs a `file://` base location next to it. |
Review Comment:
Thanks, confirmed.
`buildLoadTableResponseWithDelegationCredentials` runs on every table load, and for a federated catalog it calls `validateTableLocations` unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`, so an `hdfs://` table location is rejected at load time with a 403 even if the catalog itself could be created. I did not run an HDFS cluster end to end; the conclusion follows from the storage type prefix rules and that check.
The latest commit adds that load-time check to this row and changes the Hive row to match ("Only with a `*` wildcard in `allowedLocations`"), with a link back to this page, so the two tables agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add a documentation page for Hadoop catalog federation [polaris]
Posted by "youngrae81 (via GitHub)" <gi...@apache.org>.
youngrae81 commented on code in PR #5883:
URL: https://github.com/apache/polaris/pull/5883#discussion_r4407383072
##########
site/content/in-dev/unreleased/federation/hadoop-catalog-federation.md:
##########
@@ -0,0 +1,192 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Hadoop Catalog Federation
+type: docs
+weight: 706
+---
+
+Polaris can federate catalog operations to an Iceberg
+[Hadoop catalog](https://iceberg.apache.org/docs/latest/java-api-quickstart/#using-a-hadoop-catalog):
+a warehouse directory in which namespaces and tables are tracked by directory layout, without a
+metastore service. This lets existing file-based Iceberg tables remain where they are while Polaris
+brokers access, policies, and multi-engine connectivity.
+
+{{% alert title="Note" color="primary" %}}
+The Iceberg documentation states that the Hadoop catalog "can only be used with HDFS or similar
+file systems that support atomic rename" and that "concurrent writes with a Hadoop catalog are not
+safe with a local FS or S3". Those constraints apply unchanged when the catalog is reached through
+Polaris.
+{{% /alert %}}
+
+## Build-time enablement
+
+Unlike Hive and BigQuery federation, the Hadoop factory is part of the default server build:
+`runtime/server/build.gradle.kts` always includes `polaris-extensions-federation-hadoop`, so no
+Gradle property is needed. Standard Polaris binaries and container images can federate to Hadoop
+catalogs as soon as the feature flags below are set.
+
+## Feature configuration
+
+Enable the necessary feature flags in your `application.properties` file (or equivalent
+configuration mechanism such as environment variables or a Kubernetes ConfigMap):
+
+```properties
+# Allows both REST and HADOOP connection types
+polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["ICEBERG_REST","HADOOP"]
+
+# Allows IMPLICIT authentication, needed for Hadoop federation
+polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["OAUTH","IMPLICIT"]
+
+# Enables the federation feature itself
+polaris.features."ENABLE_CATALOG_FEDERATION"=true
+```
+
+For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris container
+(typically at `/deployment/config/application.properties`).
+
+## Runtime requirements
+
+- **Warehouse access:** The Polaris server itself lists, reads, and writes the warehouse directory.
+ There is no remote service in between, so the file system behind the `warehouse` location must be
+ reachable from the Polaris process with permission to do all three.
+- **Configuration discovery:** Polaris initializes the `HadoopCatalog` with a default Hadoop
+ `Configuration`, which reads `core-site.xml` (and `hdfs-site.xml` once the HDFS client loads)
+ from the JVM classpath. `HADOOP_CONF_DIR` is added to the classpath by the Hadoop launcher
+ scripts, which the Polaris server and image do not use, so place those files on the Polaris
+ server classpath, for example through an image layer, when the warehouse is on HDFS or needs
+ non-default client settings.
+- **Authentication:** Hadoop federation only supports `IMPLICIT` authentication, meaning Polaris
+ uses the operating-system or Kerberos identity of the running process (no stored secrets). For a
+ Kerberized HDFS, ensure the service principal holds a valid keytab/TGT before starting Polaris;
+ the [Kerberos setup example]({{% ref "hive-metastore-federation.md#kerberos-setup-example" %}})
+ for Hive federation applies here as well.
+- **Object storage access:** The server-side `HadoopCatalog` reaches an S3 warehouse with the
+ ambient AWS credentials of the Polaris process; the `roleArn` in `storageConfigInfo` is used for
+ vending credentials to clients, which for federated catalogs is governed by
+ `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING`. Read the warehouse access notes below.
+
+## Warehouse access patterns
+
+The default Polaris build ships the Hadoop client libraries and `S3FileIO`, but not Hadoop cloud
+connectors such as `hadoop-aws`. Polaris also has no HDFS storage type: `allowedLocations` and
+`default-base-location` must use a prefix accepted by one of the `FILE`, `S3`, `AZURE` or `GCS`
+storage types. Together this leads to the following patterns:
+
+| Pattern | Works with the default build | Notes |
+|--------------------------------------------------------------------------------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Local or network-mounted directory (`file:///...`) with `FILE` storage | Yes | Simplest setup for tests and single-writer deployments. `FILE` storage is intended for testing; it performs no credential vending. |
+| HDFS warehouse handled by ambient Hadoop config and process identity | Not expressible in `storageConfigInfo` | `hdfs://` is not an accepted location prefix for any storage type, and `default-base-location` must lie within an allowed location, so an HDFS warehouse cannot be described directly. Table loads from a federated catalog also check every table location against `allowedLocations` before vending credentials (unless `ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING` is `false`), so an `hdfs://` table location fails at load time too. The `*` wildcard entry (`ALLOW_WILDCARD_LOCATION=true`) lifts these location checks but still needs a `file://` base location next to it. |
+| `HadoopFileIO` with `s3a://` warehouses | Only with extra runtime packaging | `hadoop-aws` and related file system dependencies are not included in the Polaris runtime. |
+| `S3FileIO` with ambient AWS credentials | Sometimes | Set `io-impl=org.apache.iceberg.aws.s3.S3FileIO` in the catalog-level `properties`. Commits still rely on atomic rename, which S3 does not provide, so concurrent writers are unsafe. |
+| **UNSAFE!** Putting object-store credentials in catalog properties | **DO NOT USE / THIS IS UNSAFE** | Secrets placed in catalog properties are returned to authenticated catalog clients through `/config`. This **exposes passwords, access keys, session tokens, or other secrets as plain text**. |
+
+## Creating a federated catalog
+
+Use the Management API (or the Python CLI) to create an external catalog whose connection type is
+`HADOOP`. The following request registers a catalog whose tables live under
+`file:///data/warehouse/analytics`, a directory mounted into the Polaris container:
+
+```bash
+curl -X POST https://<polaris-host>/management/v1/catalogs \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "type": "EXTERNAL",
+ "name": "analytics_hadoop",
+ "storageConfigInfo": {
+ "storageType": "FILE",
+ "allowedLocations": ["file:///data/warehouse/analytics/"]
+ },
+ "properties": { "default-base-location": "file:///data/warehouse/analytics/" },
+ "connectionConfigInfo": {
+ "connectionType": "HADOOP",
+ "uri": "file:///data/warehouse/analytics/",
+ "warehouse": "file:///data/warehouse/analytics/",
+ "authenticationParameters": { "authenticationType": "IMPLICIT" }
+ }
+ }'
+```
+
+The same catalog can be created with the CLI:
+
+```bash
+polaris catalogs create analytics_hadoop \
+ --type external \
+ --storage-type file \
Review Comment:
Added a second Management API example in the latest commit: `S3` storage with `roleArn`/`region`, an `s3://` warehouse, and `io-impl=S3FileIO` in the catalog-level properties so the default build does not need `hadoop-aws`.
I kept the `file://` example first as the minimal one. HDFS is left out because an `hdfs://` warehouse cannot be expressed in `storageConfigInfo` today (see the access-pattern table).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org