You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iggy.apache.org by "mlevkov (via GitHub)" <gi...@apache.org> on 2026/03/25 18:18:12 UTC
Feature idea: Per-message HTTP header forwarding in sink connectors
GitHub user mlevkov created a discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
## Feature idea: Per-message HTTP header forwarding
*Moved from [discussion #2919](https://github.com/apache/iggy/discussions/2919#discussioncomment-16300152) at @hubcio's request.*
**Context**: While working on the HTTP sink PR (#2925), a question came up about dynamic per-message headers. Today, all HTTP headers are static (config-derived, same for every request). But there are real use cases for per-message headers.
### Use cases
- `X-Iggy-Offset` / `X-Iggy-Topic` / `X-Iggy-Partition` — routing and dedup at the HTTP endpoint without body parsing
- Forwarding Iggy user-defined message headers (`ConsumedMessage.headers`) as HTTP headers — e.g., a producer attaches `X-Correlation-ID` and the sink forwards it to the downstream endpoint
### Proposed design — additive layering
```rust
// Static config headers (pre-built once in open(), cloned per-request)
let mut request = build_request(self.method, client, &self.url)
.headers(self.request_headers.clone())
.header("content-type", content_type);
// Per-message dynamic headers (opt-in via config)
if self.include_metadata_headers {
request = request
.header("x-iggy-offset", offset.to_string())
.header("x-iggy-topic", &topic_metadata.topic)
.header("x-iggy-partition", partition_id.to_string());
}
// Forward Iggy user headers as HTTP headers (opt-in)
if self.forward_iggy_headers {
if let Some(headers) = &message.headers {
for (key, value) in headers {
request = request.header(
format!("x-iggy-{}", key.to_string_value()),
value.to_string_value(),
);
}
}
}
```
### Config surface
```toml
[plugin_config]
# Include iggy metadata (offset, topic, partition) as HTTP headers
include_metadata_headers = false
# Forward iggy user-defined message headers as HTTP headers (prefixed with x-iggy-)
forward_iggy_headers = false
```
This preserves the current pre-built `HeaderMap` optimization for static headers while layering dynamic per-message headers on top. Both options are opt-in and disabled by default.
### Design considerations
- **Batch modes**: For `ndjson` and `json_array`, per-message headers only apply if the batch contains a single message, or we'd need to pick "representative" headers from the batch (e.g., first message, or common subset). For `individual` mode it's straightforward — each message is its own HTTP request.
- **Header name conflicts**: If a user-defined Iggy header name conflicts with a static config header, the static header should win (config is explicit intent). Alternatively, we could log a warning.
- **Performance**: Adding per-message headers is a few string allocations per request. For `individual` mode at high throughput this could add up, but it's behind an opt-in flag so the default path is unchanged.
This could be a follow-up PR after the base HTTP sink (#2925) lands. Thoughts?
GitHub link: https://github.com/apache/iggy/discussions/3029
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "hubcio (via GitHub) via dev" <de...@iggy.apache.org>.
GitHub user hubcio added a comment to the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
the connectors moved to [apache/iggy-connectors](https://github.com/apache/iggy-connectors), and this discussion moved with them: https://github.com/apache/iggy-connectors/discussions/65. please continue there.
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-19524544
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315188
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "spetz (via GitHub)" <gi...@apache.org>.
GitHub user spetz added a comment to the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
I think it could be a useful feature, one thing I'd change - we could ditch `x-` prefix, and just use e.g. `iggy-offset` instead.
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16314570
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa edited a comment on the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
So we should have `Apache-Iggy-Offset, Apache-Iggy-Topic, Apache-Iggy-Partition` etc
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315200
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa added a comment to the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315200
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "mlevkov (via GitHub) via dev" <de...@iggy.apache.org>.
GitHub user mlevkov closed a discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
## Feature idea: Per-message HTTP header forwarding
*Moved from [discussion #2919](https://github.com/apache/iggy/discussions/2919#discussioncomment-16300152) at @hubcio's request.*
**Context**: While working on the HTTP sink PR (#2925), a question came up about dynamic per-message headers. Today, all HTTP headers are static (config-derived, same for every request). But there are real use cases for per-message headers.
### Use cases
- `X-Iggy-Offset` / `X-Iggy-Topic` / `X-Iggy-Partition` — routing and dedup at the HTTP endpoint without body parsing
- Forwarding Iggy user-defined message headers (`ConsumedMessage.headers`) as HTTP headers — e.g., a producer attaches `X-Correlation-ID` and the sink forwards it to the downstream endpoint
### Proposed design — additive layering
```rust
// Static config headers (pre-built once in open(), cloned per-request)
let mut request = build_request(self.method, client, &self.url)
.headers(self.request_headers.clone())
.header("content-type", content_type);
// Per-message dynamic headers (opt-in via config)
if self.include_metadata_headers {
request = request
.header("x-iggy-offset", offset.to_string())
.header("x-iggy-topic", &topic_metadata.topic)
.header("x-iggy-partition", partition_id.to_string());
}
// Forward Iggy user headers as HTTP headers (opt-in)
if self.forward_iggy_headers {
if let Some(headers) = &message.headers {
for (key, value) in headers {
request = request.header(
format!("x-iggy-{}", key.to_string_value()),
value.to_string_value(),
);
}
}
}
```
### Config surface
```toml
[plugin_config]
# Include iggy metadata (offset, topic, partition) as HTTP headers
include_metadata_headers = false
# Forward iggy user-defined message headers as HTTP headers (prefixed with x-iggy-)
forward_iggy_headers = false
```
This preserves the current pre-built `HeaderMap` optimization for static headers while layering dynamic per-message headers on top. Both options are opt-in and disabled by default.
### Design considerations
- **Batch modes**: For `ndjson` and `json_array`, per-message headers only apply if the batch contains a single message, or we'd need to pick "representative" headers from the batch (e.g., first message, or common subset). For `individual` mode it's straightforward — each message is its own HTTP request.
- **Header name conflicts**: If a user-defined Iggy header name conflicts with a static config header, the static header should win (config is explicit intent). Alternatively, we could log a warning.
- **Performance**: Adding per-message headers is a few string allocations per request. For `individual` mode at high throughput this could add up, but it's behind an opt-in flag so the default path is unchanged.
This could be a follow-up PR after the base HTTP sink (#2925) lands. Thoughts?
GitHub link: https://github.com/apache/iggy/discussions/3029
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa deleted a comment on the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315188
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa edited a comment on the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
So we should have `Apache-Iggy-Offset, Apache-Iggy-Topic, Apache-Iggy-Partition` etc or without `Apache-` prefix to save space. Just hope that they won't collide with any other headers :)
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315200
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "kparisa (via GitHub)" <gi...@apache.org>.
GitHub user kparisa edited a comment on the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
@mlevkov If we leverage OTEL, this would be the Trace-Id right?
and +1 to @spetz recommendation to drop X- prefix. In modern implementations, we don't need that as per the proposal https://datatracker.ietf.org/doc/html/rfc6648
So we should have `Apache-Iggy-Offset, Apache-Iggy-Topic, Apache-Iggy-Partition` etc or without Apache- prefix to save space. Just hope that they won't collide with any other headers :)
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16315200
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org
Re: Feature idea: Per-message HTTP header forwarding in sink connectors
Posted by "mlevkov (via GitHub)" <gi...@apache.org>.
GitHub user mlevkov added a comment to the discussion: Feature idea: Per-message HTTP header forwarding in sink connectors
Yes, likely so, or even any other header.
GitHub link: https://github.com/apache/iggy/discussions/3029#discussioncomment-16319711
----
This is an automatically sent email for dev@iggy.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@iggy.apache.org