You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "elliotsteene-swap (via GitHub)" <gi...@apache.org> on 2026/03/16 11:10:43 UTC
[I] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
elliotsteene-swap opened a new issue, #61388:
URL: https://github.com/apache/doris/issues/61388
## Feature Request: Support `header.*` Properties for REST Iceberg Catalog
### Summary
Apache Doris should support passing custom HTTP headers to the Iceberg REST catalog client via `header.<Header-Name>` catalog properties. This is part of the standard Iceberg REST catalog specification and is required to integrate with catalogs such as **GCP BigLake Metastore** that mandate custom headers for authentication and routing.
### Motivation and Use Case
#### GCP BigLake Metastore
[Google Cloud BigLake Metastore](https://cloud.google.com/bigquery/docs/iceberg-tables) exposes an Iceberg REST catalog endpoint but requires callers to include a `x-goog-user-project` header on every request to identify the billing project. Without this header, requests are rejected.
The standard way to supply this in the Iceberg ecosystem is via the `header.x-goog-user-project` catalog property, which the Iceberg REST client translates into an HTTP header on all outgoing requests.
Example of the expected Doris `CREATE CATALOG` syntax once this feature is supported:
```sql
CREATE CATALOG biglake_catalog
PROPERTIES (
'type' = 'iceberg',
'iceberg.catalog.type' = 'rest',
'iceberg.rest.uri' = 'https://biglakeapi.googleapis.com/v1/projects/<PROJECT>/locations/<REGION>/catalogs/<CATALOG>',
'warehouse' = 'projects/<PROJECT>/locations/<REGION>/catalogs/<CATALOG>',
'header.x-goog-user-project' = '<BILLING_PROJECT>'
);
```
Without `header.*` support, it is impossible to use Doris with BigLake Metastore (and any other REST catalog that relies on custom request headers for auth/routing).
### Standard Iceberg Behaviour
The `header.*` property prefix is a first-class convention in the Iceberg REST catalog client (`org.apache.iceberg.rest.RESTCatalog`). Any property whose key starts with `header.` is stripped of the prefix and injected as an HTTP header on every request the client makes.
This behaviour is documented and used across the Iceberg ecosystem:
- **Apache Iceberg Java / Spark** – `spark.sql.catalog.my_catalog.header.X-Catalog-Name=default`
- **PyIceberg** – `header.x-goog-user-project`, `header.X-Iceberg-Access-Delegation`, etc. ([docs](https://py.iceberg.apache.org/configuration/))
- **Iceberg Rust client** – recently added the same feature
### Root Cause
Doris appears to pass only an explicitly enumerated set of properties to the underlying `RESTCatalog` instance rather than forwarding all catalog properties. As a result, `header.*` keys are silently dropped before reaching the Iceberg client, even though the client itself is capable of processing them.
The fix is to ensure that all catalog properties (or at minimum those with a `header.` prefix) are forwarded to the `restProperties` map that is handed to the Iceberg `RESTCatalog`.
### Expected Behaviour
Any `header.<Name> = <value>` property specified in `CREATE CATALOG ... PROPERTIES (...)` should be forwarded to the Iceberg REST catalog client and sent as the HTTP header `<Name>: <value>` on all requests to the REST catalog endpoint.
### Alternatives Considered
- **Network-layer proxy** – injecting headers via a sidecar/reverse-proxy is possible but fragile, adds operational complexity, and should not be required for a standard protocol feature.
- **OAuth2 / SigV4** – these are separate authentication mechanisms and do not cover arbitrary header requirements such as billing-project routing.
### Environment
- Apache Doris version: 3.x / 4.x (latest)
- Feature affects: Iceberg REST catalog integration (`iceberg.catalog.type = 'rest'`)
### References
- Apache Iceberg REST catalog spec – header properties
- [PyIceberg configuration docs](https://py.iceberg.apache.org/configuration/)
- [GCP BigLake Metastore Iceberg REST API](https://cloud.google.com/bigquery/docs/iceberg-tables)
- Related Doris issue: #23671 (property pass-through for REST catalog)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "elliotsteene-swap (via GitHub)" <gi...@apache.org>.
elliotsteene-swap commented on issue #61388:
URL: https://github.com/apache/doris/issues/61388#issuecomment-4073463742
Thanks!
I think it makes sense to have some thing like:
```
CREATE CATALOG biglake_metastore PROPERTIES (
'type' = 'iceberg',
'iceberg.catalog.type' = 'rest',
<existing iceberg.rest settings>
'iceberg.rest.header.<X-header-name>' = <header value>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "Arun12-sketch (via GitHub)" <gi...@apache.org>.
Arun12-sketch commented on issue #61388:
URL: https://github.com/apache/doris/issues/61388#issuecomment-4072325437
Hi, I'd like to work on this issue. I've looked into the codebase and
identified where the catalog properties are filtered before being passed
to the RESTCatalog. I'll submit a PR with the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #61388: [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties
URL: https://github.com/apache/doris/issues/61388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #61388:
URL: https://github.com/apache/doris/issues/61388#issuecomment-5753838070
We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on issue #61388:
URL: https://github.com/apache/doris/issues/61388#issuecomment-4107173284
Thanks @Arun12-sketch!
Are you already working on this fix? We'd love to review your PR once it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [feature] Support custom HTTP headers for REST Iceberg catalog via header.* properties [doris]
Posted by "Arun12-sketch (via GitHub)" <gi...@apache.org>.
Arun12-sketch commented on issue #61388:
URL: https://github.com/apache/doris/issues/61388#issuecomment-4115376606
Hi @morningman, I’ve implemented the fix and opened a PR for this issue.
Looking forward to your review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org