[PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov opened a new pull request, #2925:
URL: https://github.com/apache/iggy/pull/2925

   ## Summary
   
   Adds a **generic HTTP sink connector** that delivers consumed Iggy stream messages to any HTTP endpoint — webhooks, Lambda functions, REST APIs, or SaaS integrations.
   
   - **4 batch modes**: `individual` (one request per message), `ndjson` (newline-delimited), `json_array` (single array), `raw` (bytes)
   - **Retry with exponential backoff**: transient error classification (429/5xx), `Retry-After` header respect, configurable max retries/delay
   - **Metadata envelope**: optional Iggy metadata wrapping (offset, timestamp, stream, topic, partition), base64 for binary payloads
   - **Partial delivery**: in `individual`/`raw` modes, continues after per-message failure with consecutive failure abort threshold
   - **47 unit tests**, zero clippy warnings
   
   ### Design
   
   Follows established connector patterns:
   
   | Pattern | Source | Applied |
   |---------|--------|---------|
   | `Option<reqwest::Client>` with lazy init in `open()` | MongoDB, ES, PostgreSQL | Connection pooling, configurable timeout |
   | `AtomicU64` counters | MongoDB | Track requests, messages, errors, retries |
   | Retry with backoff | PostgreSQL, MongoDB | Transient error classification + exponential backoff |
   | Config via serde `Deserialize` | All sinks | `HttpSinkConfig` with `Option` fields, resolved in `new()` |
   | `sink_connector!` macro | All sinks | FFI export |
   
   All retry logic lives inside `consume()` — the runtime currently discards the `Result` from `consume()` (`sink.rs:585`) and commits offsets before processing. These are known upstream issues documented in the README.
   
   ### Files
   
   ```
   core/connectors/sinks/http_sink/
   ├── Cargo.toml     # workspace deps only (no new dependencies)
   ├── config.toml    # example configuration
   ├── README.md      # usage, config reference, known limitations
   └── src/lib.rs     # implementation + 47 unit tests (1757 lines)
   ```
   
   ### Configuration
   
   ```toml
   [plugin_config]
   url = "https://api.example.com/ingest"
   method = "POST"
   batch_mode = "ndjson"
   timeout = "30s"
   max_retries = 3
   retry_delay = "1s"
   retry_backoff_multiplier = 2.0
   include_metadata = true
   
   [plugin_config.headers]
   Authorization = "Bearer my-token"
   ```
   
   See [README](core/connectors/sinks/http_sink/README.md) for full configuration reference and examples.
   
   ### Context
   
   This was discussed in [GitHub Discussion #2901](https://github.com/apache/iggy/discussions/2901) with positive feedback from the maintainers. The design follows all patterns established by the existing 6 sink connectors.
   
   ## Test plan
   
   - [x] 47 unit tests covering config resolution, payload conversion, metadata envelope, retry delay computation, transient status classification, UUID formatting, TOML deserialization, `open()` validation
   - [x] Zero clippy warnings (`cargo clippy -p iggy_connector_http_sink -- -D warnings`)
   - [x] Compiles as both `cdylib` and `lib`
   - [ ] Manual test with `iggy-connectors` runtime (Try It section in README)
   - [ ] Integration test with mock HTTP server (future follow-up)
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2936683260


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,

Review Comment:
   `iggy_timestamp` (server-assigned) vs `iggy_origin_timestamp` (producer-supplied) -- the difference isn't obvious from the names. these are new public api surface. once shipped, the names become a contract. consider `iggy_server_timestamp` or at least document the distinction prominently.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(

Review Comment:
   iggy messages can carry user-defined `headers: Option<HashMap<HeaderKey, HeaderValue>>`. the http sink ignores them entirely -- not forwarded as http headers, not in the metadata envelope, not logged. the elasticsearch sink includes them. if a producer attaches routing/tracing headers, they just vanish. either forward them (e.g. `iggy_headers` in the envelope) or document as unsupported.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {
+    match value {
+        simd_json::OwnedValue::Static(s) => match s {
+            simd_json::StaticNode::Null => serde_json::Value::Null,
+            simd_json::StaticNode::Bool(b) => serde_json::Value::Bool(*b),
+            simd_json::StaticNode::I64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::U64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::F64(n) => serde_json::Number::from_f64(*n)
+                .map(serde_json::Value::Number)
+                .unwrap_or(serde_json::Value::Null),
+        },
+        simd_json::OwnedValue::String(s) => serde_json::Value::String(s.to_string()),
+        simd_json::OwnedValue::Array(arr) => {
+            serde_json::Value::Array(arr.iter().map(owned_value_to_serde_json).collect())
+        }
+        simd_json::OwnedValue::Object(obj) => {
+            let map: serde_json::Map<String, serde_json::Value> = obj
+                .iter()
+                .map(|(k, v)| (k.to_string(), owned_value_to_serde_json(v)))
+                .collect();
+            serde_json::Value::Object(map)
+        }
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as a UUID-style hex string (8-4-4-4-12).
+/// This is positional formatting only — no RFC 4122 version/variant bits are set.
+/// Downstream consumers should treat this as an opaque identifier, not a standards-compliant UUID.
+fn format_u128_as_uuid(id: u128) -> String {

Review Comment:
   `format_u128_as_uuid()` produces strings that look exactly like uuids but have no version/variant bits. downstream consumers in go/java/python/c# will parse this with uuid libraries and get validation failures or make wrong assumptions. options: (a) use the `uuid` crate with version 8 custom, (b) emit plain hex without dashes, or (c) rename the field to something like `iggy_message_id_hex` so nobody mistakes it for a uuid.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(

Review Comment:
   also `send_raw` at lines 866-961.
   
   both do the same thing: iterate messages, build body, check size, send with retry, track consecutive failures, circuit-break at `MAX_CONSECUTIVE_FAILURES`, report partial delivery. only the body-building step differs. extract a shared `send_per_message()` parameterized by a closure.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {

Review Comment:
   linear scan on the hottest path. default list is 4 elements so it's not a disaster, but a `HashSet` or sorted vec with binary search would be better.



##########
core/connectors/sinks/http_sink/config.toml:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+type = "sink"
+key = "http"
+enabled = true
+version = 0
+name = "HTTP sink"
+path = "../../target/release/libiggy_connector_http_sink"
+verbose = false
+
+[[streams]]
+stream = "my_stream"
+topics = ["my_topic"]
+schema = "json"
+batch_length = 50
+poll_interval = "100ms"
+consumer_group = "http_sink_group"
+
+[plugin_config]
+# Required — target URL for HTTP requests.
+url = "https://api.example.com/ingest"
+
+# HTTP method (default: POST). Valid: GET, HEAD, POST, PUT, PATCH, DELETE.
+method = "POST"
+
+# Request timeout (default: 30s).
+timeout = "30s"
+
+# Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+max_payload_size_bytes = 10485760
+
+# Payload formatting mode (default: individual).
+# - "individual": one HTTP request per message
+# - "ndjson": newline-delimited JSON, all messages in one request
+# - "json_array": JSON array of messages in one request
+# - "raw": raw bytes, individual requests only
+batch_mode = "ndjson"
+
+# Include Iggy metadata envelope (default: true).
+include_metadata = true
+
+# Include message checksum in metadata (default: false).
+include_checksum = false
+
+# Include origin timestamp in metadata (default: false).
+include_origin_timestamp = false
+
+# Health check — opt-in, disabled by default.
+# Many endpoints (Lambda, API Gateway) don't support HEAD/OPTIONS.
+health_check_enabled = false
+health_check_method = "HEAD"
+
+# Retry configuration.
+max_retries = 3
+retry_delay = "1s"
+retry_backoff_multiplier = 2.0
+max_retry_delay = "30s"
+
+# HTTP status codes considered successful (default: [200, 201, 202, 204]).
+success_status_codes = [200, 201, 202, 204]
+
+# TLS — accept invalid certs (default: false). Use only for development.
+tls_danger_accept_invalid_certs = false
+
+# Connection pool — max idle connections per host (default: 10).
+max_connections = 10
+
+# Verbose request/response logging (default: false).
+verbose_logging = false
+
+# Custom HTTP headers. Replace placeholder values with real credentials.
+# Do not commit actual secrets — use environment variable overrides for production.
+[plugin_config.headers]
+Authorization = "Bearer my-secret-token"

Review Comment:
   `Authorization = "Bearer my-secret-token"` may trigger secret-scanning tools. use `"Bearer <YOUR_TOKEN_HERE>"` instead.



##########
core/integration/tests/connectors/http/http_sink.rs:
##########
@@ -0,0 +1,1119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! HTTP Sink Connector: Integration Tests
+//!
+//! **Purpose**: End-to-end validation of the HTTP sink connector — messages flow from
+//! Iggy streams through the connector runtime, get transformed by the sink plugin, and
+//! arrive at a real HTTP endpoint where we verify format, headers, metadata, and content.
+//!
+//! ## Connector Architecture
+//!
+//! The HTTP sink runs inside the Iggy connector runtime as a dynamically loaded plugin:
+//!
+//! ```text
+//! ┌──────────────┐    ┌──────────────────────┐    ┌──────────────────┐
+//! │  Test Code   │    │  Connector Runtime   │    │    WireMock      │
+//! │              │    │                      │    │                  │
+//! │ send_messages├───►│  iggy-server (poll)  │    │  /__admin/       │
+//! │              │    │        │             │    │  (verify reqs)   │
+//! │              │    │  ┌─────▼──────────┐  │    │                  │
+//! │ wait_for_    │    │  │ HTTP Sink      │  │    │  /ingest         │
+//! │ requests ◄───┼────┤  │ (.so/.dylib)   ├──┼───►│  (accept POST)   │
+//! │              │    │  └────────────────┘  │    │                  │
+//! └──────────────┘    └──────────────────────┘    └──────────────────┘
+//! ```
+//!
+//! **Key components**:
+//! 1. **iggy-server**: Stores messages in streams/topics, serves them to consumers
+//! 2. **Connector runtime**: `iggy-connectors` binary, loads the HTTP sink `.so`/`.dylib`
+//!    plugin via FFI, polls topics, calls `iggy_sink_consume()` per batch
+//! 3. **HTTP sink plugin**: Transforms messages into HTTP requests (4 batch modes),
+//!    applies metadata envelope, retries on failure
+//! 4. **WireMock**: Docker container accepting all POSTs to `/ingest`, recording
+//!    requests for later verification via `/__admin/requests`
+//!
+//! **Runtime model**: 1 process = 1 config = 1 plugin. The runtime reads `config.toml`,
+//! loads the plugin binary, iterates `for topic in stream.topics`, and spawns one
+//! `tokio::spawn` task per topic. Each task creates an `IggyConsumer` and polls
+//! sequentially — `consume()` is awaited before the next poll.
+//!
+//! See `setup_sink_consumers()` and `spawn_consume_tasks()` in `runtime/src/sink.rs`.
+//!
+//! ## What These Tests Validate
+//!
+//! **Test 1 — Individual Mode**: Each message becomes a separate HTTP POST with
+//! metadata envelope (`{metadata: {...}, payload: {...}}`). Validates envelope
+//! structure, content type, and per-message delivery.
+//!
+//! **Test 2 — NDJSON Batch Mode**: All messages arrive in one HTTP request as
+//! newline-delimited JSON. Validates line count, per-line envelope structure,
+//! and `application/x-ndjson` content type.
+//!
+//! **Test 3 — JSON Array Batch Mode**: All messages arrive in one HTTP request
+//! as a JSON array. Validates array length, per-item envelope structure, and
+//! `application/json` content type.
+//!
+//! **Test 4 — Raw Mode**: Each message sent as raw bytes without metadata envelope.
+//! Validates `application/octet-stream` content type and absence of envelope wrapper.
+//!
+//! **Test 5 — Metadata Disabled**: Individual mode with `include_metadata=false`.
+//! Validates that the bare payload arrives without the `{metadata, payload}` wrapper.
+//!
+//! **Test 6 — Sequential Offsets**: Sends 5 messages and verifies `iggy_offset` values
+//! in metadata are contiguous (each offset = previous + 1). Validates that the
+//! connector preserves Iggy's offset ordering through the HTTP delivery pipeline.
+//!
+//! **Test 7 — Multi-Topic**: One connector consuming from two topics on the same
+//! stream. Validates that `iggy_topic` metadata correctly identifies the source topic,
+//! and that messages from both topics arrive at the shared endpoint. Exercises the
+//! runtime's per-topic task spawning (`spawn_consume_tasks()` in `runtime/src/sink.rs`).
+//!
+//! ## Test Infrastructure
+//!
+//! **Full-Stack Integration** (all components are real — no mocks):
+//! - **iggy-server**: Started by `#[iggy_harness]` macro, in-process
+//! - **Connector runtime**: Started by harness with `connectors_runtime(config_path = ...)`
+//! - **HTTP sink plugin**: Built from `core/connectors/sinks/http_sink/` (must be compiled)
+//! - **WireMock**: Docker container (`wiremock/wiremock:3.13.2`) via testcontainers
+//! - **Test fixtures**: `HttpSink*Fixture` structs configure batch mode, metadata, topics
+//!   via environment variables that override `config.toml` fields
+//!
+//! **Fixture Architecture**:
+//! Each fixture implements `TestFixture` trait, returning `connectors_runtime_envs()` that
+//! override the plugin config. The base configuration (`HttpSinkIndividualFixture::base_envs`)
+//! sets URL, method, timeout, retries, stream/topic, and schema. Specialized fixtures
+//! (NDJSON, JSON array, raw, no-metadata, multi-topic) override specific fields.
+//!
+//! **WireMock Container**:
+//! Accepts all POSTs to `/ingest` (via `accept-ingest.json` mapping). Exposes
+//! `/__admin/requests` for polling received requests. The container uses a bind mount
+//! for mappings and a health check wait strategy for readiness.
+//!
+//! **Seed Data**:
+//! `seeds::connector_stream` creates the stream (`test_stream`) and first topic
+//! (`test_topic`). The multi-topic test creates a second topic inline to avoid
+//! polluting the shared harness with HTTP-sink-specific constants.
+//!
+//! **Configuration** (`tests/connectors/http/sink.toml`):
+//! ```toml
+//! [connectors]
+//! config_type = "local"
+//! config_dir = "../connectors/sinks/http_sink"
+//! ```
+//! Environment variables override `config.toml` fields at runtime. Convention:
+//! `IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_<FIELD>` (e.g., `..._BATCH_MODE=ndjson`).
+//!
+//! ## Running Tests
+//!
+//! ```bash
+//! # Prerequisites: Docker running, HTTP sink plugin compiled
+//! cargo build -p iggy_connector_http_sink
+//!
+//! # Run all HTTP sink integration tests
+//! cargo test -p integration --test connectors -- http_sink --nocapture
+//!
+//! # Run a specific test
+//! cargo test -p integration --test connectors -- individual_json_messages --nocapture
+//!
+//! # Run with test isolation (sequential)
+//! cargo test -p integration --test connectors -- http_sink --test-threads=1 --nocapture
+//! ```
+//!
+//! ## Success Criteria
+//!
+//! - **All 4 batch modes**: Messages arrive in correct format (individual, ndjson, json_array, raw)
+//! - **Metadata envelope**: Present when `include_metadata=true`, absent when `false`
+//! - **Content types**: `application/json` (individual/json_array), `application/x-ndjson`,
+//!   `application/octet-stream` (raw)
+//! - **Offset ordering**: Sequential, contiguous offsets in metadata
+//! - **Multi-topic routing**: `iggy_topic` metadata matches source topic for each message
+//! - **Message counts**: Exact match between sent and received message counts
+//!
+//! ## Related Documentation
+//!
+//! - **HTTP Sink README**: `core/connectors/sinks/http_sink/README.md` — Config reference,
+//!   deployment patterns, retry strategy, connection pooling, message flow
+//! - **Connector Runtime**: `runtime/src/sink.rs` — `setup_sink_consumers()`,
+//!   `spawn_consume_tasks()`, `consume_messages()`, FFI boundary
+//! - **SDK Macro**: `sdk/src/sink.rs` — `sink_connector!` macro, `SinkContainer`, DashMap
+//! - **Fixtures**: `tests/connectors/fixtures/http/` — WireMock container, fixture structs
+//! - **PR**: https://github.com/apache/iggy/pull/2925
+//! - **Discussion**: https://github.com/apache/iggy/discussions/2919
+//!
+//! ## Known Limitations
+//!
+//! 1. **FFI return value ignored**: The runtime's `process_messages()` discards `consume()`'s
+//!    `i32` return code. Errors are logged by the sink but invisible to the runtime.
+//!    See [#2927](https://github.com/apache/iggy/issues/2927).
+//! 2. **Offsets committed before processing**: `PollingMessages` auto-commit strategy commits
+//!    offsets before `consume()`. Combined with (1), effective guarantee is at-most-once.
+//!    See [#2928](https://github.com/apache/iggy/issues/2928).
+//!
+//! ## Test History
+//!
+//! - **2026-03-10**: Initial test suite — 6 tests covering all batch modes, metadata toggle,
+//!   and sequential offset verification.
+//! - **2026-03-11**: Added multi-topic test (Test 7). Initially used shared harness seed
+//!   (`connector_multi_topic_stream`) with `TOPIC_2` constant in `seeds.rs`. Removed during
+//!   code review remediation — second topic now created inline to keep harness generic.
+//! - **2026-03-12**: Code review rounds 3+4 (double-review protocol). Fixed: magic string
+//!   match arms replaced with constants (M9), harness pollution removed (H1).
+
+use super::TEST_MESSAGE_COUNT;
+use crate::connectors::fixtures::{
+    HttpSinkIndividualFixture, HttpSinkJsonArrayFixture, HttpSinkMultiTopicFixture,
+    HttpSinkNdjsonFixture, HttpSinkNoMetadataFixture, HttpSinkRawFixture,
+};
+use bytes::Bytes;
+use iggy::prelude::{IggyMessage, Partitioning};
+use iggy_binary_protocol::MessageClient;
+use iggy_common::Identifier;
+use integration::harness::seeds;
+use integration::iggy_harness;
+
+// ============================================================================
+// Test 1: Individual Batch Mode
+// ============================================================================
+
+/// Test 1: Individual JSON Messages Delivered as Separate HTTP POSTs

Review Comment:
   per-test doc comments are 30-50 lines each. module-level doc is 158 lines. trim per-test docs to 3-5 lines; the module-level comment already covers architecture well enough.



##########
core/integration/tests/connectors/fixtures/http/container.rs:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use integration::harness::TestBinaryError;
+use std::time::Duration;
+use testcontainers_modules::testcontainers::core::WaitFor::Healthcheck;
+use testcontainers_modules::testcontainers::core::wait::HealthWaitStrategy;
+use testcontainers_modules::testcontainers::core::{IntoContainerPort, Mount};
+use testcontainers_modules::testcontainers::runners::AsyncRunner;
+use testcontainers_modules::testcontainers::{ContainerAsync, GenericImage, ImageExt};
+use tokio::time::sleep;
+use tracing::info;
+
+const WIREMOCK_IMAGE: &str = "wiremock/wiremock";
+const WIREMOCK_TAG: &str = "3.13.2";
+const WIREMOCK_PORT: u16 = 8080;
+
+pub(super) const DEFAULT_TEST_STREAM: &str = "test_stream";
+pub(super) const DEFAULT_TEST_TOPIC: &str = "test_topic";
+pub(super) const DEFAULT_TEST_TOPIC_2: &str = "test_topic_2";
+
+pub(super) const DEFAULT_POLL_ATTEMPTS: usize = 100;
+pub(super) const DEFAULT_POLL_INTERVAL_MS: u64 = 100;
+
+// HTTP sink env vars follow the convention: IGGY_CONNECTORS_SINK_HTTP_<SECTION>_<FIELD>
+pub(super) const ENV_SINK_PATH: &str = "IGGY_CONNECTORS_SINK_HTTP_PATH";
+pub(super) const ENV_SINK_STREAMS_0_STREAM: &str = "IGGY_CONNECTORS_SINK_HTTP_STREAMS_0_STREAM";
+pub(super) const ENV_SINK_STREAMS_0_TOPICS: &str = "IGGY_CONNECTORS_SINK_HTTP_STREAMS_0_TOPICS";
+pub(super) const ENV_SINK_STREAMS_0_SCHEMA: &str = "IGGY_CONNECTORS_SINK_HTTP_STREAMS_0_SCHEMA";
+pub(super) const ENV_SINK_STREAMS_0_CONSUMER_GROUP: &str =
+    "IGGY_CONNECTORS_SINK_HTTP_STREAMS_0_CONSUMER_GROUP";
+
+// plugin_config fields
+pub(super) const ENV_SINK_URL: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_URL";
+pub(super) const ENV_SINK_BATCH_MODE: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_BATCH_MODE";
+pub(super) const ENV_SINK_INCLUDE_METADATA: &str =
+    "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_INCLUDE_METADATA";
+pub(super) const ENV_SINK_METHOD: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_METHOD";
+pub(super) const ENV_SINK_TIMEOUT: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_TIMEOUT";
+pub(super) const ENV_SINK_MAX_RETRIES: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_MAX_RETRIES";
+pub(super) const ENV_SINK_RETRY_DELAY: &str = "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_RETRY_DELAY";
+pub(super) const ENV_SINK_VERBOSE_LOGGING: &str =
+    "IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_VERBOSE_LOGGING";
+
+/// WireMock container for HTTP sink integration tests.
+///
+/// Provides a real HTTP endpoint that accepts requests and exposes an admin API
+/// for verifying received requests at `/__admin/requests`.
+pub struct HttpSinkWireMockContainer {
+    #[allow(dead_code)]
+    container: ContainerAsync<GenericImage>,
+    /// Base URL of the WireMock container (e.g., `http://localhost:32768`).
+    pub(super) base_url: String,
+}
+
+impl HttpSinkWireMockContainer {
+    pub(super) async fn start() -> Result<Self, TestBinaryError> {
+        let current_dir = std::env::current_dir().map_err(|e| TestBinaryError::FixtureSetup {
+            fixture_type: "HttpSinkWireMockContainer".to_string(),
+            message: format!("Failed to get current dir: {e}"),
+        })?;
+
+        let container = GenericImage::new(WIREMOCK_IMAGE, WIREMOCK_TAG)
+            .with_exposed_port(WIREMOCK_PORT.tcp())
+            .with_wait_for(Healthcheck(HealthWaitStrategy::default()))
+            .with_mount(Mount::bind_mount(
+                current_dir
+                    .join("tests/connectors/http/wiremock/mappings")
+                    .to_string_lossy()
+                    .to_string(),
+                "/home/wiremock/mappings",
+            ))
+            .start()
+            .await
+            .map_err(|e| TestBinaryError::FixtureSetup {
+                fixture_type: "HttpSinkWireMockContainer".to_string(),
+                message: format!("Failed to start container: {e}"),
+            })?;
+
+        let host = container
+            .get_host()
+            .await
+            .map_err(|e| TestBinaryError::FixtureSetup {
+                fixture_type: "HttpSinkWireMockContainer".to_string(),
+                message: format!("Failed to get host: {e}"),
+            })?;
+
+        let host_port = container
+            .get_host_port_ipv4(WIREMOCK_PORT)
+            .await
+            .map_err(|e| TestBinaryError::FixtureSetup {
+                fixture_type: "HttpSinkWireMockContainer".to_string(),
+                message: format!("Failed to get port: {e}"),
+            })?;
+
+        let base_url = format!("http://{host}:{host_port}");
+        info!("HTTP sink WireMock container available at {base_url}");
+
+        Ok(Self {
+            container,
+            base_url,
+        })
+    }
+
+    /// Query WireMock's admin API and return all received requests.
+    pub async fn get_received_requests(&self) -> Result<Vec<WireMockRequest>, TestBinaryError> {
+        let url = format!("{}/__admin/requests", self.base_url);
+        let response = reqwest::get(&url)
+            .await
+            .map_err(|e| TestBinaryError::InvalidState {
+                message: format!("Failed to query WireMock admin API: {e}"),
+            })?;
+
+        let body: serde_json::Value =
+            response
+                .json()
+                .await
+                .map_err(|e| TestBinaryError::InvalidState {
+                    message: format!("Failed to parse WireMock admin response: {e}"),
+                })?;
+
+        let requests = body["requests"]
+            .as_array()
+            .unwrap_or(&vec![])

Review Comment:
   use a static empty slice or refactor to avoid the allocation. it's test code so it doesn't really matter, but it's sloppy.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);

Review Comment:
   called inside `send_with_retry`'s loop (line 434). for individual mode with 50 messages and 3 retries, that's 200 header rebuilds per poll cycle. build a `HeaderMap` once in `open()` and reuse it.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {
+    match value {
+        simd_json::OwnedValue::Static(s) => match s {
+            simd_json::StaticNode::Null => serde_json::Value::Null,
+            simd_json::StaticNode::Bool(b) => serde_json::Value::Bool(*b),
+            simd_json::StaticNode::I64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::U64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::F64(n) => serde_json::Number::from_f64(*n)
+                .map(serde_json::Value::Number)
+                .unwrap_or(serde_json::Value::Null),
+        },
+        simd_json::OwnedValue::String(s) => serde_json::Value::String(s.to_string()),
+        simd_json::OwnedValue::Array(arr) => {
+            serde_json::Value::Array(arr.iter().map(owned_value_to_serde_json).collect())
+        }
+        simd_json::OwnedValue::Object(obj) => {
+            let map: serde_json::Map<String, serde_json::Value> = obj
+                .iter()
+                .map(|(k, v)| (k.to_string(), owned_value_to_serde_json(v)))
+                .collect();
+            serde_json::Value::Object(map)
+        }
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as a UUID-style hex string (8-4-4-4-12).
+/// This is positional formatting only — no RFC 4122 version/variant bits are set.
+/// Downstream consumers should treat this as an opaque identifier, not a standards-compliant UUID.
+fn format_u128_as_uuid(id: u128) -> String {
+    let hex = format!("{:032x}", id);
+    format!(
+        "{}-{}-{}-{}-{}",
+        &hex[0..8],
+        &hex[8..12],
+        &hex[12..16],
+        &hex[16..20],
+        &hex[20..32],
+    )
+}
+
+/// Truncate a response body string for log output, respecting UTF-8 char boundaries.
+fn truncate_response(body: &str, max_len: usize) -> &str {
+    if body.len() <= max_len {
+        body
+    } else {
+        // Find the last valid UTF-8 char boundary at or before max_len
+        let end = body.floor_char_boundary(max_len);
+        &body[..end]
+    }
+}
+
+#[async_trait]
+impl Sink for HttpSink {
+    async fn open(&mut self) -> Result<(), Error> {
+        // Validate success_status_codes — empty would cause every response to be treated as failure
+        if self.success_status_codes.is_empty() {
+            return Err(Error::InitError(
+                "success_status_codes must not be empty — would cause retry storms against healthy endpoints".to_string(),
+            ));
+        }
+        for &code in &self.success_status_codes {
+            if !(200..=599).contains(&code) {
+                return Err(Error::InitError(format!(
+                    "Invalid status code {} in success_status_codes — must be 200-599",
+                    code,
+                )));
+            }
+        }
+
+        // Validate URL
+        if self.url.is_empty() {
+            return Err(Error::InitError(
+                "HTTP sink URL is empty — 'url' is required in [plugin_config]".to_string(),
+            ));
+        }
+        match reqwest::Url::parse(&self.url) {
+            Ok(parsed) => {
+                let scheme = parsed.scheme();
+                if scheme != "http" && scheme != "https" {
+                    return Err(Error::InitError(format!(
+                        "HTTP sink URL scheme '{}' is not allowed — only 'http' and 'https' are supported (url: '{}')",
+                        scheme, self.url,
+                    )));
+                }
+            }
+            Err(e) => {
+                return Err(Error::InitError(format!(
+                    "HTTP sink URL '{}' is not a valid URL: {}",
+                    self.url, e,
+                )));
+            }
+        }
+
+        // Warn if user supplied a Content-Type header — it will be overridden by batch_mode.
+        if self
+            .headers
+            .keys()
+            .any(|k| k.eq_ignore_ascii_case("content-type"))
+        {
+            warn!(
+                "HTTP sink ID: {} — custom 'Content-Type' header in [headers] is ignored. \
+                 Content-Type is set by batch_mode ({:?} -> '{}'). \
+                 Remove it from [headers] to silence this warning.",
+                self.id,
+                self.batch_mode,
+                self.content_type(),
+            );
+        }
+
+        // Validate custom headers — fail fast rather than per-request errors
+        for (key, value) in &self.headers {
+            reqwest::header::HeaderName::from_bytes(key.as_bytes())
+                .map_err(|e| Error::InitError(format!("Invalid header name '{}': {}", key, e)))?;
+            reqwest::header::HeaderValue::from_str(value).map_err(|e| {
+                Error::InitError(format!("Invalid header value for '{}': {}", key, e))
+            })?;
+        }
+
+        // Build the HTTP client with config-derived settings
+        self.client = Some(self.build_client()?);
+
+        // Optional health check — uses same success_status_codes and headers as consume()
+        if self.health_check_enabled {
+            let client = self.client.as_ref().expect("client just built");
+            let mut health_request = build_request(self.health_check_method, client, &self.url);
+            for (key, value) in &self.headers {
+                health_request = health_request.header(key, value);
+            }
+
+            let response = health_request.send().await.map_err(|e| {
+                Error::Connection(format!("Health check failed for URL '{}': {}", self.url, e))
+            })?;
+
+            let status = response.status();
+            if !self.success_status_codes.contains(&status.as_u16()) {
+                return Err(Error::Connection(format!(
+                    "Health check returned status {} (not in success_status_codes {:?}) for URL '{}'",
+                    status.as_u16(),
+                    self.success_status_codes,
+                    self.url,
+                )));
+            }
+
+            info!(
+                "HTTP sink ID: {} — health check passed (status {})",
+                self.id,
+                status.as_u16()
+            );
+        }
+
+        info!(
+            "Opened HTTP sink connector with ID: {} for URL: {} (method: {:?}, \
+             batch_mode: {:?}, timeout: {:?}, max_retries: {})",
+            self.id, self.url, self.method, self.batch_mode, self.timeout, self.max_retries,
+        );
+        Ok(())
+    }
+
+    /// Deliver messages to the configured HTTP endpoint.
+    ///
+    /// **Runtime note**: The connector runtime's `process_messages()` in `runtime/src/sink.rs` currently discards the `Result`
+    /// returned by `consume()`. All retry logic lives inside this method — returning `Err`
+    /// does not trigger a runtime-level retry. This is a known upstream issue.
+    async fn consume(

Review Comment:
   individual mode with `batch_length=50`, `max_retries=3`, `max_retry_delay=30s` could take up to `50 * 4 * 30s = 6000s` worst case. the `MAX_CONSECUTIVE_FAILURES` circuit breaker helps but a pathological fail-succeed-fail-succeed pattern bypasses it. at least document the worst-case latency.



##########
core/integration/tests/connectors/http/http_sink.rs:
##########
@@ -0,0 +1,1119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! HTTP Sink Connector: Integration Tests
+//!
+//! **Purpose**: End-to-end validation of the HTTP sink connector — messages flow from
+//! Iggy streams through the connector runtime, get transformed by the sink plugin, and
+//! arrive at a real HTTP endpoint where we verify format, headers, metadata, and content.
+//!
+//! ## Connector Architecture
+//!
+//! The HTTP sink runs inside the Iggy connector runtime as a dynamically loaded plugin:
+//!
+//! ```text
+//! ┌──────────────┐    ┌──────────────────────┐    ┌──────────────────┐
+//! │  Test Code   │    │  Connector Runtime   │    │    WireMock      │
+//! │              │    │                      │    │                  │
+//! │ send_messages├───►│  iggy-server (poll)  │    │  /__admin/       │
+//! │              │    │        │             │    │  (verify reqs)   │
+//! │              │    │  ┌─────▼──────────┐  │    │                  │
+//! │ wait_for_    │    │  │ HTTP Sink      │  │    │  /ingest         │
+//! │ requests ◄───┼────┤  │ (.so/.dylib)   ├──┼───►│  (accept POST)   │
+//! │              │    │  └────────────────┘  │    │                  │
+//! └──────────────┘    └──────────────────────┘    └──────────────────┘
+//! ```
+//!
+//! **Key components**:
+//! 1. **iggy-server**: Stores messages in streams/topics, serves them to consumers
+//! 2. **Connector runtime**: `iggy-connectors` binary, loads the HTTP sink `.so`/`.dylib`
+//!    plugin via FFI, polls topics, calls `iggy_sink_consume()` per batch
+//! 3. **HTTP sink plugin**: Transforms messages into HTTP requests (4 batch modes),
+//!    applies metadata envelope, retries on failure
+//! 4. **WireMock**: Docker container accepting all POSTs to `/ingest`, recording
+//!    requests for later verification via `/__admin/requests`
+//!
+//! **Runtime model**: 1 process = 1 config = 1 plugin. The runtime reads `config.toml`,
+//! loads the plugin binary, iterates `for topic in stream.topics`, and spawns one
+//! `tokio::spawn` task per topic. Each task creates an `IggyConsumer` and polls
+//! sequentially — `consume()` is awaited before the next poll.
+//!
+//! See `setup_sink_consumers()` and `spawn_consume_tasks()` in `runtime/src/sink.rs`.
+//!
+//! ## What These Tests Validate
+//!
+//! **Test 1 — Individual Mode**: Each message becomes a separate HTTP POST with
+//! metadata envelope (`{metadata: {...}, payload: {...}}`). Validates envelope
+//! structure, content type, and per-message delivery.
+//!
+//! **Test 2 — NDJSON Batch Mode**: All messages arrive in one HTTP request as
+//! newline-delimited JSON. Validates line count, per-line envelope structure,
+//! and `application/x-ndjson` content type.
+//!
+//! **Test 3 — JSON Array Batch Mode**: All messages arrive in one HTTP request
+//! as a JSON array. Validates array length, per-item envelope structure, and
+//! `application/json` content type.
+//!
+//! **Test 4 — Raw Mode**: Each message sent as raw bytes without metadata envelope.
+//! Validates `application/octet-stream` content type and absence of envelope wrapper.
+//!
+//! **Test 5 — Metadata Disabled**: Individual mode with `include_metadata=false`.
+//! Validates that the bare payload arrives without the `{metadata, payload}` wrapper.
+//!
+//! **Test 6 — Sequential Offsets**: Sends 5 messages and verifies `iggy_offset` values
+//! in metadata are contiguous (each offset = previous + 1). Validates that the
+//! connector preserves Iggy's offset ordering through the HTTP delivery pipeline.
+//!
+//! **Test 7 — Multi-Topic**: One connector consuming from two topics on the same
+//! stream. Validates that `iggy_topic` metadata correctly identifies the source topic,
+//! and that messages from both topics arrive at the shared endpoint. Exercises the
+//! runtime's per-topic task spawning (`spawn_consume_tasks()` in `runtime/src/sink.rs`).
+//!
+//! ## Test Infrastructure
+//!
+//! **Full-Stack Integration** (all components are real — no mocks):
+//! - **iggy-server**: Started by `#[iggy_harness]` macro, in-process
+//! - **Connector runtime**: Started by harness with `connectors_runtime(config_path = ...)`
+//! - **HTTP sink plugin**: Built from `core/connectors/sinks/http_sink/` (must be compiled)
+//! - **WireMock**: Docker container (`wiremock/wiremock:3.13.2`) via testcontainers
+//! - **Test fixtures**: `HttpSink*Fixture` structs configure batch mode, metadata, topics
+//!   via environment variables that override `config.toml` fields
+//!
+//! **Fixture Architecture**:
+//! Each fixture implements `TestFixture` trait, returning `connectors_runtime_envs()` that
+//! override the plugin config. The base configuration (`HttpSinkIndividualFixture::base_envs`)
+//! sets URL, method, timeout, retries, stream/topic, and schema. Specialized fixtures
+//! (NDJSON, JSON array, raw, no-metadata, multi-topic) override specific fields.
+//!
+//! **WireMock Container**:
+//! Accepts all POSTs to `/ingest` (via `accept-ingest.json` mapping). Exposes
+//! `/__admin/requests` for polling received requests. The container uses a bind mount
+//! for mappings and a health check wait strategy for readiness.
+//!
+//! **Seed Data**:
+//! `seeds::connector_stream` creates the stream (`test_stream`) and first topic
+//! (`test_topic`). The multi-topic test creates a second topic inline to avoid
+//! polluting the shared harness with HTTP-sink-specific constants.
+//!
+//! **Configuration** (`tests/connectors/http/sink.toml`):
+//! ```toml
+//! [connectors]
+//! config_type = "local"
+//! config_dir = "../connectors/sinks/http_sink"
+//! ```
+//! Environment variables override `config.toml` fields at runtime. Convention:
+//! `IGGY_CONNECTORS_SINK_HTTP_PLUGIN_CONFIG_<FIELD>` (e.g., `..._BATCH_MODE=ndjson`).
+//!
+//! ## Running Tests
+//!
+//! ```bash
+//! # Prerequisites: Docker running, HTTP sink plugin compiled
+//! cargo build -p iggy_connector_http_sink
+//!
+//! # Run all HTTP sink integration tests
+//! cargo test -p integration --test connectors -- http_sink --nocapture
+//!
+//! # Run a specific test
+//! cargo test -p integration --test connectors -- individual_json_messages --nocapture
+//!
+//! # Run with test isolation (sequential)
+//! cargo test -p integration --test connectors -- http_sink --test-threads=1 --nocapture
+//! ```
+//!
+//! ## Success Criteria
+//!
+//! - **All 4 batch modes**: Messages arrive in correct format (individual, ndjson, json_array, raw)
+//! - **Metadata envelope**: Present when `include_metadata=true`, absent when `false`
+//! - **Content types**: `application/json` (individual/json_array), `application/x-ndjson`,
+//!   `application/octet-stream` (raw)
+//! - **Offset ordering**: Sequential, contiguous offsets in metadata
+//! - **Multi-topic routing**: `iggy_topic` metadata matches source topic for each message
+//! - **Message counts**: Exact match between sent and received message counts
+//!
+//! ## Related Documentation
+//!
+//! - **HTTP Sink README**: `core/connectors/sinks/http_sink/README.md` — Config reference,
+//!   deployment patterns, retry strategy, connection pooling, message flow
+//! - **Connector Runtime**: `runtime/src/sink.rs` — `setup_sink_consumers()`,
+//!   `spawn_consume_tasks()`, `consume_messages()`, FFI boundary
+//! - **SDK Macro**: `sdk/src/sink.rs` — `sink_connector!` macro, `SinkContainer`, DashMap
+//! - **Fixtures**: `tests/connectors/fixtures/http/` — WireMock container, fixture structs
+//! - **PR**: https://github.com/apache/iggy/pull/2925
+//! - **Discussion**: https://github.com/apache/iggy/discussions/2919
+//!
+//! ## Known Limitations
+//!
+//! 1. **FFI return value ignored**: The runtime's `process_messages()` discards `consume()`'s
+//!    `i32` return code. Errors are logged by the sink but invisible to the runtime.
+//!    See [#2927](https://github.com/apache/iggy/issues/2927).
+//! 2. **Offsets committed before processing**: `PollingMessages` auto-commit strategy commits
+//!    offsets before `consume()`. Combined with (1), effective guarantee is at-most-once.
+//!    See [#2928](https://github.com/apache/iggy/issues/2928).
+//!
+//! ## Test History
+//!
+//! - **2026-03-10**: Initial test suite — 6 tests covering all batch modes, metadata toggle,
+//!   and sequential offset verification.
+//! - **2026-03-11**: Added multi-topic test (Test 7). Initially used shared harness seed
+//!   (`connector_multi_topic_stream`) with `TOPIC_2` constant in `seeds.rs`. Removed during
+//!   code review remediation — second topic now created inline to keep harness generic.
+//! - **2026-03-12**: Code review rounds 3+4 (double-review protocol). Fixed: magic string
+//!   match arms replaced with constants (M9), harness pollution removed (H1).
+
+use super::TEST_MESSAGE_COUNT;
+use crate::connectors::fixtures::{
+    HttpSinkIndividualFixture, HttpSinkJsonArrayFixture, HttpSinkMultiTopicFixture,
+    HttpSinkNdjsonFixture, HttpSinkNoMetadataFixture, HttpSinkRawFixture,
+};
+use bytes::Bytes;
+use iggy::prelude::{IggyMessage, Partitioning};
+use iggy_binary_protocol::MessageClient;
+use iggy_common::Identifier;
+use integration::harness::seeds;
+use integration::iggy_harness;
+
+// ============================================================================
+// Test 1: Individual Batch Mode
+// ============================================================================
+
+/// Test 1: Individual JSON Messages Delivered as Separate HTTP POSTs
+///
+/// **Purpose**: Validates that `batch_mode=individual` sends one HTTP request per Iggy
+/// message, with each request containing the full metadata envelope.
+///
+/// **Behavior Under Test**:
+/// When configured with `batch_mode=individual`, the HTTP sink's `send_individual()` method
+/// iterates over each message in the consumed batch and calls `send_with_retry()` for each
+/// one independently. The metadata envelope wraps each message with Iggy context:
+/// ```json
+/// {
+///   "metadata": { "iggy_stream": "...", "iggy_topic": "...", "iggy_offset": N, ... },
+///   "payload": { ... original message ... }
+/// }
+/// ```
+///
+/// **Why This Matters**:
+/// Individual mode is the simplest and most compatible delivery pattern. It works with any
+/// HTTP endpoint that accepts POST requests — no special parsing logic needed on the receiver.
+/// Each message is independently retryable: if message 2 of 5 fails, only message 2 is retried.
+/// This is the default batch mode and the most common deployment pattern.
+///
+/// **Test Flow**:
+/// 1. Send 3 JSON messages to Iggy (`test_stream`/`test_topic`, partition 0)
+/// 2. Wait for WireMock to receive exactly 3 HTTP requests
+/// 3. Verify each request: POST method, `/ingest` URL
+/// 4. Verify each body: `metadata` and `payload` fields present
+/// 5. Verify metadata: `iggy_stream`, `iggy_topic`, `iggy_offset` fields present
+/// 6. Verify content type: `application/json`
+///
+/// **Key Validations**:
+/// - Request count = message count (1:1 mapping)
+/// - Metadata envelope structure is correct
+/// - Content-Type header is `application/json`
+/// - All 3 standard metadata fields present (`iggy_stream`, `iggy_topic`, `iggy_offset`)
+///
+/// **Related Code**:
+/// - `send_individual()` in `sinks/http_sink/src/lib.rs` — per-message delivery loop
+/// - `build_envelope()` in `sinks/http_sink/src/lib.rs` — metadata envelope construction
+/// - `HttpSinkIndividualFixture` in `fixtures/http/sink.rs` — base fixture with env overrides
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn individual_json_messages_delivered_as_separate_posts(
+    harness: &TestHarness,
+    fixture: HttpSinkIndividualFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 3 JSON messages with distinct payloads
+    let json_payloads: Vec<serde_json::Value> = vec![
+        serde_json::json!({"name": "Alice", "age": 30}),
+        serde_json::json!({"name": "Bob", "score": 99}),
+        serde_json::json!({"name": "Carol", "active": true}),
+    ];
+
+    let mut messages: Vec<IggyMessage> = json_payloads
+        .iter()
+        .enumerate()
+        .map(|(i, payload)| {
+            let bytes = serde_json::to_vec(payload).expect("Failed to serialize");
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(bytes))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for WireMock to receive all 3 individual HTTP requests
+    // In individual mode, each message becomes a separate HTTP request.
+    let requests = fixture
+        .container()
+        .wait_for_requests(TEST_MESSAGE_COUNT)
+        .await
+        .expect("WireMock did not receive expected number of requests");
+
+    assert_eq!(
+        requests.len(),
+        TEST_MESSAGE_COUNT,
+        "Expected exactly {TEST_MESSAGE_COUNT} individual requests, got {}",
+        requests.len()
+    );
+
+    // Step 4: Verify each request has correct method, URL, and envelope structure
+    for req in &requests {
+        assert_eq!(req.method, "POST", "Expected POST method");
+        assert_eq!(req.url, "/ingest", "Expected /ingest URL");
+
+        let body = req.body_as_json().expect("Body should be valid JSON");
+
+        // Metadata envelope: {metadata: {...}, payload: {...}}
+        assert!(
+            body.get("metadata").is_some(),
+            "Expected metadata envelope in individual mode, got: {body}"
+        );
+        assert!(
+            body.get("payload").is_some(),
+            "Expected payload field in individual mode, got: {body}"
+        );
+
+        // Verify standard metadata fields from Iggy context
+        let metadata = &body["metadata"];
+        assert!(
+            metadata.get("iggy_stream").is_some(),
+            "Expected iggy_stream in metadata"
+        );
+        assert!(
+            metadata.get("iggy_topic").is_some(),
+            "Expected iggy_topic in metadata"
+        );
+        assert!(
+            metadata.get("iggy_offset").is_some(),
+            "Expected iggy_offset in metadata"
+        );
+    }
+
+    // Step 5: Verify content type header
+    let ct = requests[0]
+        .header("Content-Type")
+        .expect("Content-Type header must be present");
+    assert!(
+        ct.contains("application/json"),
+        "Expected application/json content type, got: {ct}"
+    );
+}
+
+// ============================================================================
+// Test 2: NDJSON Batch Mode
+// ============================================================================
+
+/// Test 2: NDJSON Batch Mode — All Messages in One Newline-Delimited Request
+///
+/// **Purpose**: Validates that `batch_mode=ndjson` combines all messages into a single
+/// HTTP request with newline-delimited JSON body (`application/x-ndjson`).
+///
+/// **Behavior Under Test**:
+/// The HTTP sink's `send_ndjson()` method serializes each message as a JSON envelope,
+/// joins them with `\n`, and sends the result as a single HTTP request. This mode is
+/// optimal for endpoints that accept streaming JSON (e.g., Elasticsearch `_bulk` API,
+/// cloud logging services, data lake ingestion). The `send_batch_body()` helper handles
+/// the post-send accounting (error counting, skip warnings) shared with `send_json_array`.
+///
+/// **Why This Matters**:
+/// NDJSON reduces HTTP overhead from N requests to 1 request for a batch of N messages.
+/// For high-throughput streams (thousands of messages per second), this can reduce
+/// connection overhead by orders of magnitude. Individual serialization failures are
+/// skipped (with error counting) rather than aborting the entire batch — partial delivery
+/// is preferred over total failure.
+///
+/// **Test Flow**:
+/// 1. Send 3 JSON event messages to Iggy
+/// 2. Wait for WireMock to receive exactly 1 HTTP request
+/// 3. Split response body by newlines — expect 3 lines
+/// 4. Parse each line as JSON, verify `metadata` and `payload` fields
+/// 5. Verify content type: `application/x-ndjson`
+///
+/// **Key Validations**:
+/// - Single HTTP request (all messages batched)
+/// - Line count = message count
+/// - Each NDJSON line is valid JSON with metadata envelope
+/// - Content-Type is `application/x-ndjson`
+///
+/// **Related Code**:
+/// - `send_ndjson()` in `sinks/http_sink/src/lib.rs` — NDJSON serialization and size check
+/// - `send_batch_body()` in `sinks/http_sink/src/lib.rs` — shared batch delivery + accounting
+/// - `HttpSinkNdjsonFixture` in `fixtures/http/sink.rs` — overrides `BATCH_MODE=ndjson`
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn ndjson_messages_delivered_as_single_request(
+    harness: &TestHarness,
+    fixture: HttpSinkNdjsonFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 3 JSON event messages
+    let json_payloads: Vec<serde_json::Value> = vec![
+        serde_json::json!({"event": "login", "user": 1}),
+        serde_json::json!({"event": "click", "user": 2}),
+        serde_json::json!({"event": "logout", "user": 3}),
+    ];
+
+    let mut messages: Vec<IggyMessage> = json_payloads
+        .iter()
+        .enumerate()
+        .map(|(i, payload)| {
+            let bytes = serde_json::to_vec(payload).expect("Failed to serialize");
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(bytes))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for single NDJSON request (all messages batched into one)
+    let requests = fixture
+        .container()
+        .wait_for_requests(1)
+        .await
+        .expect("WireMock did not receive NDJSON request");
+
+    let req = &requests[0];
+    assert_eq!(req.method, "POST", "Expected POST method");
+    assert_eq!(req.url, "/ingest", "Expected /ingest URL");
+
+    // Step 4: Parse NDJSON body — each line is a separate JSON envelope
+    let lines: Vec<&str> = req.body.trim().lines().collect();
+    assert_eq!(
+        lines.len(),
+        TEST_MESSAGE_COUNT,
+        "Expected {TEST_MESSAGE_COUNT} NDJSON lines, got {}",
+        lines.len()
+    );
+
+    for (i, line) in lines.iter().enumerate() {
+        let parsed: serde_json::Value =
+            serde_json::from_str(line).unwrap_or_else(|e| panic!("NDJSON line {i} invalid: {e}"));
+        assert!(
+            parsed.get("metadata").is_some(),
+            "Expected metadata in NDJSON line {i}"
+        );
+        assert!(
+            parsed.get("payload").is_some(),
+            "Expected payload in NDJSON line {i}"
+        );
+    }
+
+    // Step 5: Verify NDJSON content type
+    let ct = req
+        .header("Content-Type")
+        .expect("Content-Type header must be present");
+    assert!(
+        ct.contains("application/x-ndjson"),
+        "Expected application/x-ndjson content type, got: {ct}"
+    );
+}
+
+// ============================================================================
+// Test 3: JSON Array Batch Mode
+// ============================================================================
+
+/// Test 3: JSON Array Batch Mode — All Messages as a Single JSON Array
+///
+/// **Purpose**: Validates that `batch_mode=json_array` combines all messages into a single
+/// HTTP request with a JSON array body (`[{envelope1}, {envelope2}, ...]`).
+///
+/// **Behavior Under Test**:
+/// The HTTP sink's `send_json_array()` method builds envelope structs for each message,
+/// collects them into a `Vec`, and serializes the entire vector as a JSON array via
+/// `serde_json::to_vec()`. Like NDJSON, individual serialization failures are skipped.
+/// The whole-batch serialization (the final `to_vec` call) is a separate failure point —
+/// if it fails, all successfully-built envelopes are counted as errors.
+///
+/// **Why This Matters**:
+/// JSON array mode is compatible with APIs that expect a standard JSON array (e.g., REST
+/// bulk endpoints, webhook aggregators). Unlike NDJSON, the entire body is a single valid
+/// JSON document, which simplifies parsing on the receiver side. The trade-off is that the
+/// entire body must fit in memory as a single allocation.
+///
+/// **Test Flow**:
+/// 1. Send 3 JSON messages to Iggy (order, payment, refund events)
+/// 2. Wait for WireMock to receive exactly 1 HTTP request
+/// 3. Parse body as JSON array, verify array length = 3
+/// 4. Verify each array item has `metadata` and `payload` fields
+/// 5. Verify content type: `application/json`
+///
+/// **Key Validations**:
+/// - Single HTTP request (all messages batched)
+/// - Body is a valid JSON array
+/// - Array length = message count
+/// - Each item has metadata envelope
+/// - Content-Type is `application/json`
+///
+/// **Related Code**:
+/// - `send_json_array()` in `sinks/http_sink/src/lib.rs` — array serialization + size check
+/// - `send_batch_body()` in `sinks/http_sink/src/lib.rs` — shared batch delivery + accounting
+/// - `HttpSinkJsonArrayFixture` in `fixtures/http/sink.rs` — overrides `BATCH_MODE=json_array`
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn json_array_messages_delivered_as_single_request(
+    harness: &TestHarness,
+    fixture: HttpSinkJsonArrayFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 3 JSON messages representing different event types
+    let json_payloads: Vec<serde_json::Value> = vec![
+        serde_json::json!({"id": 1, "type": "order"}),
+        serde_json::json!({"id": 2, "type": "payment"}),
+        serde_json::json!({"id": 3, "type": "refund"}),
+    ];
+
+    let mut messages: Vec<IggyMessage> = json_payloads
+        .iter()
+        .enumerate()
+        .map(|(i, payload)| {
+            let bytes = serde_json::to_vec(payload).expect("Failed to serialize");
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(bytes))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for single JSON array request (all messages in one body)
+    let requests = fixture
+        .container()
+        .wait_for_requests(1)
+        .await
+        .expect("WireMock did not receive JSON array request");
+
+    let req = &requests[0];
+    assert_eq!(req.method, "POST", "Expected POST method");
+    assert_eq!(req.url, "/ingest", "Expected /ingest URL");
+
+    // Step 4: Parse body as JSON array and verify structure
+    let body = req.body_as_json().expect("Body should be valid JSON");
+    assert!(body.is_array(), "Expected JSON array body, got: {body}");
+
+    let arr = body.as_array().unwrap();
+    assert_eq!(
+        arr.len(),
+        TEST_MESSAGE_COUNT,
+        "Expected {TEST_MESSAGE_COUNT} items in JSON array, got {}",
+        arr.len()
+    );
+
+    for (i, item) in arr.iter().enumerate() {
+        assert!(
+            item.get("metadata").is_some(),
+            "Expected metadata in array item {i}"
+        );
+        assert!(
+            item.get("payload").is_some(),
+            "Expected payload in array item {i}"
+        );
+    }
+
+    // Step 5: Verify JSON content type
+    let ct = req
+        .header("Content-Type")
+        .expect("Content-Type header must be present");
+    assert!(
+        ct.contains("application/json"),
+        "Expected application/json content type, got: {ct}"
+    );
+}
+
+// ============================================================================
+// Test 4: Raw Batch Mode
+// ============================================================================
+
+/// Test 4: Raw Binary Messages Delivered Without Metadata Envelope
+///
+/// **Purpose**: Validates that `batch_mode=raw` sends each message as raw bytes in a
+/// separate HTTP request, without the metadata envelope wrapper.
+///
+/// **Behavior Under Test**:
+/// The HTTP sink's `send_raw()` method extracts raw bytes from each message payload
+/// via `try_into_vec()` and sends them directly as the HTTP body. No JSON serialization,
+/// no metadata envelope — the body is exactly the bytes that were published to Iggy.
+/// This mode is intended for binary protocols (Protobuf, FlatBuffers) or when the
+/// receiver expects unmodified passthrough.
+///
+/// **Why This Matters**:
+/// Raw mode enables the HTTP sink to forward arbitrary binary data — protocol buffers,
+/// Avro records, compressed payloads, or any format the receiver understands. The connector
+/// acts as a transparent bridge between Iggy and the HTTP endpoint. The `include_metadata`
+/// config is ignored in raw mode (metadata requires JSON serialization which contradicts
+/// raw byte passthrough).
+///
+/// **Test Flow**:
+/// 1. Send 3 raw byte messages to Iggy (plain text for verification simplicity)
+/// 2. Wait for WireMock to receive exactly 3 HTTP requests (1:1, like individual)
+/// 3. Verify each request: POST method, `/ingest` URL
+/// 4. Verify body does NOT contain metadata envelope
+/// 5. Verify content type: `application/octet-stream`
+///
+/// **Key Validations**:
+/// - Request count = message count (raw is always 1:1)
+/// - No metadata envelope in body (raw bytes only)
+/// - Content-Type is `application/octet-stream`
+///
+/// **Related Code**:
+/// - `send_raw()` in `sinks/http_sink/src/lib.rs` — raw byte extraction and delivery
+/// - `content_type()` in `sinks/http_sink/src/lib.rs` — returns `application/octet-stream` for raw
+/// - `HttpSinkRawFixture` in `fixtures/http/sink.rs` — overrides `BATCH_MODE=raw`, `SCHEMA=raw`
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn raw_binary_messages_delivered_without_envelope(
+    harness: &TestHarness,
+    fixture: HttpSinkRawFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 3 raw byte messages
+    let raw_payloads: Vec<Vec<u8>> = vec![
+        b"plain text message".to_vec(),
+        b"another raw payload".to_vec(),
+        b"third raw message".to_vec(),
+    ];
+
+    let mut messages: Vec<IggyMessage> = raw_payloads
+        .iter()
+        .enumerate()
+        .map(|(i, payload)| {
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(payload.clone()))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for all 3 raw HTTP requests (raw mode is always 1:1)
+    let requests = fixture
+        .container()
+        .wait_for_requests(TEST_MESSAGE_COUNT)
+        .await
+        .expect("WireMock did not receive expected raw requests");
+
+    assert_eq!(
+        requests.len(),
+        TEST_MESSAGE_COUNT,
+        "Expected exactly {TEST_MESSAGE_COUNT} raw requests, got {}",
+        requests.len()
+    );
+
+    // Step 4: Verify raw mode — no metadata envelope
+    for req in &requests {
+        assert_eq!(req.method, "POST", "Expected POST method");
+        assert_eq!(req.url, "/ingest", "Expected /ingest URL");
+
+        // Raw mode: body is raw bytes, NOT a JSON envelope.
+        // If the body happens to parse as JSON, it must NOT have "metadata" key.
+        if let Ok(json) = req.body_as_json() {
+            assert!(
+                json.get("metadata").is_none(),
+                "Raw mode should not include metadata envelope"
+            );
+        }
+    }
+
+    // Step 5: Verify raw content type
+    let ct = requests[0]
+        .header("Content-Type")
+        .expect("Content-Type header must be present");
+    assert!(
+        ct.contains("application/octet-stream"),
+        "Expected application/octet-stream for raw mode, got: {ct}"
+    );
+}
+
+// ============================================================================
+// Test 5: Metadata Disabled
+// ============================================================================
+
+/// Test 5: Metadata Disabled — Bare Payload Without Envelope
+///
+/// **Purpose**: Validates that `include_metadata=false` sends the original message payload
+/// directly as the HTTP body, without the `{metadata, payload}` envelope wrapper.
+///
+/// **Behavior Under Test**:
+/// When `include_metadata=false`, the `build_envelope()` method is skipped and the
+/// serialized payload JSON is sent directly. For a message containing `{"key": "value1"}`,
+/// the HTTP body is exactly `{"key": "value1"}` — not `{"metadata": {...}, "payload": {"key": "value1"}}`.
+///
+/// **Why This Matters**:
+/// Many webhook receivers and REST APIs expect a specific JSON schema and cannot handle
+/// unexpected wrapper fields. Disabling metadata allows the HTTP sink to act as a transparent
+/// JSON forwarder. This is the correct setting when the receiver already has its own
+/// deduplication/ordering mechanism and doesn't need Iggy's stream/topic/offset context.
+///
+/// **Test Flow**:
+/// 1. Send 3 simple JSON messages to Iggy
+/// 2. Wait for WireMock to receive all 3 requests
+/// 3. Verify each body: NO `metadata` field present
+/// 4. Verify each body: original `key` field present at top level
+///
+/// **Key Validations**:
+/// - No `metadata` field in body (envelope disabled)
+/// - Original payload fields at top level (not nested under `payload`)
+///
+/// **Related Code**:
+/// - `consume()` in `sinks/http_sink/src/lib.rs` — conditional envelope based on `include_metadata`
+/// - `HttpSinkNoMetadataFixture` in `fixtures/http/sink.rs` — overrides `INCLUDE_METADATA=false`
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn metadata_disabled_sends_bare_payload(
+    harness: &TestHarness,
+    fixture: HttpSinkNoMetadataFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 3 simple JSON messages
+    let json_payloads: Vec<serde_json::Value> = vec![
+        serde_json::json!({"key": "value1"}),
+        serde_json::json!({"key": "value2"}),
+        serde_json::json!({"key": "value3"}),
+    ];
+
+    let mut messages: Vec<IggyMessage> = json_payloads
+        .iter()
+        .enumerate()
+        .map(|(i, payload)| {
+            let bytes = serde_json::to_vec(payload).expect("Failed to serialize");
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(bytes))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for WireMock to receive all requests
+    let requests = fixture
+        .container()
+        .wait_for_requests(TEST_MESSAGE_COUNT)
+        .await
+        .expect("WireMock did not receive requests");
+
+    // Step 4: Verify bare payload — no metadata wrapper
+    for (i, req) in requests.iter().enumerate() {
+        let body = req
+            .body_as_json()
+            .unwrap_or_else(|e| panic!("Request {i} body should be valid JSON: {e}"));
+
+        // Without metadata, the body IS the payload — no wrapping
+        assert!(
+            body.get("metadata").is_none(),
+            "Expected no metadata envelope when include_metadata=false, got: {body}"
+        );
+
+        // The original payload fields should be at the top level
+        assert!(
+            body.get("key").is_some(),
+            "Expected bare payload with 'key' field, got: {body}"
+        );
+    }
+}
+
+// ============================================================================
+// Test 6: Sequential Offset Verification
+// ============================================================================
+
+/// Test 6: Individual Messages Have Sequential Contiguous Offsets
+///
+/// **Purpose**: Validates that `iggy_offset` values in metadata are contiguous (each
+/// offset = previous + 1), proving the connector preserves Iggy's message ordering
+/// through the entire delivery pipeline.
+///
+/// **Behavior Under Test**:
+/// Each message published to an Iggy topic partition receives a monotonically increasing
+/// offset. The HTTP sink includes this offset in the metadata envelope as `iggy_offset`.
+/// This test verifies that the sink faithfully reproduces these offsets without gaps,
+/// reordering, or duplication — critical for consumers that use offsets for deduplication
+/// or ordering guarantees.
+///
+/// **Why This Matters**:
+/// Offset integrity is the foundation for exactly-once processing at the application level.
+/// If offsets arrive out of order or with gaps, downstream consumers cannot reliably detect
+/// duplicates or missing messages. A broken offset chain could indicate a bug in the
+/// connector's message handling, a race condition in multi-topic task scheduling, or a
+/// fundamental issue with how the runtime passes messages to the plugin.
+///
+/// **Test Flow**:
+/// 1. Send 5 JSON messages to Iggy (more than default 3 to better validate ordering)
+/// 2. Wait for WireMock to receive all 5 requests
+/// 3. Extract `iggy_offset` from each request's metadata
+/// 4. Sort offsets (delivery order may differ from publish order)
+/// 5. Verify offsets are contiguous: each offset = previous + 1
+///
+/// **Key Validations**:
+/// - All 5 messages delivered
+/// - Offsets are contiguous (no gaps)
+/// - Offsets use sliding window check (`windows(2)`) — works regardless of starting offset
+///
+/// **Related Code**:
+/// - `build_envelope()` in `sinks/http_sink/src/lib.rs` — writes `iggy_offset` from
+///   `ConsumedMessage.offset`
+/// - Iggy server assigns offsets sequentially per partition
+#[iggy_harness(
+    server(connectors_runtime(config_path = "tests/connectors/http/sink.toml")),
+    seed = seeds::connector_stream
+)]
+async fn individual_messages_have_sequential_offsets(
+    harness: &TestHarness,
+    fixture: HttpSinkIndividualFixture,
+) {
+    let client = harness.root_client().await.unwrap();
+    let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap();
+    let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap();
+
+    // Step 1: Build 5 messages (more than default 3 to better test ordering)
+    let mut messages: Vec<IggyMessage> = (0..5)
+        .map(|i| {
+            let payload =
+                serde_json::to_vec(&serde_json::json!({"idx": i})).expect("Failed to serialize");
+            IggyMessage::builder()
+                .id((i + 1) as u128)
+                .payload(Bytes::from(payload))
+                .build()
+                .expect("Failed to build message")
+        })
+        .collect();
+
+    // Step 2: Publish messages to Iggy
+    client
+        .send_messages(
+            &stream_id,
+            &topic_id,
+            &Partitioning::partition_id(0),
+            &mut messages,
+        )
+        .await
+        .expect("Failed to send messages");
+
+    // Step 3: Wait for all 5 requests
+    let requests = fixture
+        .container()
+        .wait_for_requests(5)
+        .await
+        .expect("WireMock did not receive all 5 requests");
+
+    // Step 4: Extract offsets from metadata
+    // Note: offsets may not start at 0 if the seed already published messages.
+    let mut offsets: Vec<i64> = requests
+        .iter()
+        .enumerate()
+        .map(|(i, r)| {
+            let body = r
+                .body_as_json()
+                .unwrap_or_else(|e| panic!("Request {i} body is not valid JSON: {e}"));
+            body["metadata"]["iggy_offset"].as_i64().unwrap_or_else(|| {
+                panic!(
+                    "Request {i} missing or non-integer iggy_offset in metadata: {}",
+                    body["metadata"]
+                )
+            })
+        })
+        .collect();
+
+    // Step 5: Sort and verify contiguous offsets (delivery order may vary)
+    offsets.sort();
+    assert_eq!(
+        offsets.len(),
+        5,
+        "Expected 5 offsets, got {}",
+        offsets.len()
+    );
+
+    for window in offsets.windows(2) {
+        assert_eq!(
+            window[1],
+            window[0] + 1,
+            "Offsets must be contiguous: got {} then {}",
+            window[0],
+            window[1]
+        );
+    }
+}
+
+// ============================================================================
+// Test 7: Multi-Topic Delivery
+// ============================================================================
+
+/// Second topic name for the multi-topic test. Defined locally to avoid
+/// polluting the shared harness seeds with HTTP-sink-specific constants.
+///
+/// **Design Decision**: The shared `seeds.rs` module provides generic seed functions
+/// (`connector_stream`, `mcp_standard`) used by all connector types. Adding HTTP-sink-specific
+/// constants there would create coupling. Instead, this test creates the second topic inline
+/// after the seed runs, keeping the harness generic. See code review finding H1.
+const TEST_TOPIC_2: &str = seeds::names::TOPIC_2;

Review Comment:
   just use `seeds::names::TOPIC_2` directly at the usage sites.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {
+    match value {
+        simd_json::OwnedValue::Static(s) => match s {
+            simd_json::StaticNode::Null => serde_json::Value::Null,
+            simd_json::StaticNode::Bool(b) => serde_json::Value::Bool(*b),
+            simd_json::StaticNode::I64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::U64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::F64(n) => serde_json::Number::from_f64(*n)
+                .map(serde_json::Value::Number)
+                .unwrap_or(serde_json::Value::Null),
+        },
+        simd_json::OwnedValue::String(s) => serde_json::Value::String(s.to_string()),
+        simd_json::OwnedValue::Array(arr) => {
+            serde_json::Value::Array(arr.iter().map(owned_value_to_serde_json).collect())
+        }
+        simd_json::OwnedValue::Object(obj) => {
+            let map: serde_json::Map<String, serde_json::Value> = obj
+                .iter()
+                .map(|(k, v)| (k.to_string(), owned_value_to_serde_json(v)))
+                .collect();
+            serde_json::Value::Object(map)
+        }
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as a UUID-style hex string (8-4-4-4-12).
+/// This is positional formatting only — no RFC 4122 version/variant bits are set.
+/// Downstream consumers should treat this as an opaque identifier, not a standards-compliant UUID.
+fn format_u128_as_uuid(id: u128) -> String {
+    let hex = format!("{:032x}", id);

Review Comment:
   two `format!` calls. a stack-allocated approach via `write!` to a `[u8; 36]` would be zero-alloc.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {

Review Comment:
   exact copy of the same function in the elasticsearch sink. should live in `iggy_connector_sdk` as a shared utility. bug fixes for future `simd_json::OwnedValue` variants would need to be applied to both independently otherwise.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {

Review Comment:
   also at lines 727, 799, 880.
   
   `consume()` takes `messages: Vec<ConsumedMessage>` by value, but all send methods iterate via `&messages` and `.clone()` each payload. for `Payload::Raw(Vec<u8>)`, `FlatBuffer(Vec<u8>)`, and `Json(OwnedValue)` this is a full heap copy per message. fix is trivial -- change to `for message in messages` (consuming iterator) since `payload_to_json()` and `try_into_vec()` already take `Payload` by value.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {
+    match value {
+        simd_json::OwnedValue::Static(s) => match s {
+            simd_json::StaticNode::Null => serde_json::Value::Null,
+            simd_json::StaticNode::Bool(b) => serde_json::Value::Bool(*b),
+            simd_json::StaticNode::I64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::U64(n) => serde_json::Value::Number((*n).into()),
+            simd_json::StaticNode::F64(n) => serde_json::Number::from_f64(*n)
+                .map(serde_json::Value::Number)
+                .unwrap_or(serde_json::Value::Null),
+        },
+        simd_json::OwnedValue::String(s) => serde_json::Value::String(s.to_string()),
+        simd_json::OwnedValue::Array(arr) => {
+            serde_json::Value::Array(arr.iter().map(owned_value_to_serde_json).collect())
+        }
+        simd_json::OwnedValue::Object(obj) => {
+            let map: serde_json::Map<String, serde_json::Value> = obj
+                .iter()
+                .map(|(k, v)| (k.to_string(), owned_value_to_serde_json(v)))
+                .collect();
+            serde_json::Value::Object(map)
+        }
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as a UUID-style hex string (8-4-4-4-12).
+/// This is positional formatting only — no RFC 4122 version/variant bits are set.
+/// Downstream consumers should treat this as an opaque identifier, not a standards-compliant UUID.
+fn format_u128_as_uuid(id: u128) -> String {
+    let hex = format!("{:032x}", id);
+    format!(
+        "{}-{}-{}-{}-{}",
+        &hex[0..8],
+        &hex[8..12],
+        &hex[12..16],
+        &hex[16..20],
+        &hex[20..32],
+    )
+}
+
+/// Truncate a response body string for log output, respecting UTF-8 char boundaries.
+fn truncate_response(body: &str, max_len: usize) -> &str {
+    if body.len() <= max_len {
+        body
+    } else {
+        // Find the last valid UTF-8 char boundary at or before max_len
+        let end = body.floor_char_boundary(max_len);
+        &body[..end]
+    }
+}
+
+#[async_trait]
+impl Sink for HttpSink {
+    async fn open(&mut self) -> Result<(), Error> {
+        // Validate success_status_codes — empty would cause every response to be treated as failure
+        if self.success_status_codes.is_empty() {
+            return Err(Error::InitError(
+                "success_status_codes must not be empty — would cause retry storms against healthy endpoints".to_string(),
+            ));
+        }
+        for &code in &self.success_status_codes {
+            if !(200..=599).contains(&code) {

Review Comment:
   also interacts with `is_transient_status` at line 370.
   
   configuring `success_status_codes = [200, 503]` means a 503 is treated as success (success check at line 463 runs before transient check at line 481), which quietly disables retry for that code. at minimum warn if success codes overlap with the transient set `[429, 500, 502, 503, 504]`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2990413035


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()

Review Comment:
   Alternatively, we could offer both in the envelope — a valid UUID for downstream compatibility and the raw value for lossless correlation:
   
   ```json
   {
     "iggy_id": "01234567-89ab-8def-8123-456789abcdef",
     "iggy_id_raw": 5940264471900273711
   }
   ```
   
   This way downstream UUID libraries work with `iggy_id`, and anyone needing to trace back to the exact Iggy message uses `iggy_id_raw`. The raw field would be opt-in (similar to `include_checksum` / `include_origin_timestamp`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986314295


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,

Review Comment:
   Actually, wouldn't that be simpler if `client` was just a field used directly, instead of being passed every time?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r3014695861


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {

Review Comment:
   the HTTP sink uses `iggy_offset`, `iggy_partition_id`, etc. the elasticsearch sink uses `_iggy_offset`, `_iggy_partition` (leading underscore, different field name for partition). both conventions are defensible - underscore prefix is an ES convention, unprefixed is more standard for JSON APIs - but the divergence should be a deliberate decision rather than accidental.
   
   i know that this not good principles to touch many components in single PR... but my eyes hurt. maybe worth a follow-up to unify the convention?



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,
+                body.len(),
+            );
+        }
+
+        self.send_attempts.fetch_add(1, Ordering::Relaxed);
+
+        let response = build_request(self.method, client, &self.url)
+            .headers(headers.clone())
+            .header("content-type", content_type)
+            .body(body)
+            .send()
+            .await
+            .map_err(|e| {
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                error!(
+                    "HTTP sink ID: {} — request to {} failed after middleware retries: {:#}",
+                    self.id, self.url, e
+                );
+                Error::HttpRequestFailed(format!("HTTP {} — {}", self.url, e))
+            })?;
+
+        let status = response.status();
+        // success_status_codes is checked in BOTH the retry strategy (to stop retrying)
+        // AND here (to classify the final response). Both must use the same set.
+        if self.success_status_codes.contains(&status.as_u16()) {
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — success (status {})",
+                    self.id,
+                    status.as_u16()
+                );
+            }
+            self.record_success();
+            return Ok(());
+        }
+
+        // Non-success status after middleware exhausted retries — read body for diagnostics
+        let response_body = match response.text().await {
+            Ok(body) => body,
+            Err(e) => format!("<body read error: {}>", e),
+        };
+
+        error!(
+            "HTTP sink ID: {} — request failed (status {}). Response: {}",
+            self.id,
+            status.as_u16(),
+            truncate_response(&response_body, 500),
+        );
+        self.errors_count.fetch_add(1, Ordering::Relaxed);
+        Err(Error::HttpRequestFailed(format!(
+            "HTTP {} — status: {}",
+            self.url,
+            status.as_u16()
+        )))
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    self.batch_mode,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self.send_with_retry(Bytes::from(body), content_type).await {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, self.batch_mode, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id,
+                    self.batch_mode,
+                    delivered,
+                    total,
+                    http_failures,
+                    serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(messages, self.batch_mode.content_type(), |mut message| {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = self.payload_to_json(payload)?;
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json)?;
+            serde_json::to_vec(&envelope)
+                .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+        })
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(&self, body: Bytes, count: u64, skipped: u64) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(body, self.batch_mode.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);

Review Comment:
   this adds `count - 1` to `errors_count`, relying on the implicit invariant that `send_with_retry()` always adds exactly 1 on failure (at line 499 or 534). the comment at line 685 documents this, which is good, but the coupling is fragile - if `send_with_retry` is ever modified to count differently, accounting silently breaks. a `debug_assert` on the delta or a single accounting site would make this more robust.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,
+                body.len(),
+            );
+        }
+
+        self.send_attempts.fetch_add(1, Ordering::Relaxed);
+
+        let response = build_request(self.method, client, &self.url)
+            .headers(headers.clone())
+            .header("content-type", content_type)
+            .body(body)
+            .send()
+            .await
+            .map_err(|e| {
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                error!(
+                    "HTTP sink ID: {} — request to {} failed after middleware retries: {:#}",
+                    self.id, self.url, e
+                );
+                Error::HttpRequestFailed(format!("HTTP {} — {}", self.url, e))
+            })?;
+
+        let status = response.status();
+        // success_status_codes is checked in BOTH the retry strategy (to stop retrying)
+        // AND here (to classify the final response). Both must use the same set.
+        if self.success_status_codes.contains(&status.as_u16()) {
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — success (status {})",
+                    self.id,
+                    status.as_u16()
+                );
+            }
+            self.record_success();
+            return Ok(());
+        }
+
+        // Non-success status after middleware exhausted retries — read body for diagnostics
+        let response_body = match response.text().await {
+            Ok(body) => body,
+            Err(e) => format!("<body read error: {}>", e),
+        };
+
+        error!(
+            "HTTP sink ID: {} — request failed (status {}). Response: {}",
+            self.id,
+            status.as_u16(),
+            truncate_response(&response_body, 500),
+        );
+        self.errors_count.fetch_add(1, Ordering::Relaxed);
+        Err(Error::HttpRequestFailed(format!(
+            "HTTP {} — status: {}",
+            self.url,
+            status.as_u16()
+        )))
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    self.batch_mode,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self.send_with_retry(Bytes::from(body), content_type).await {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, self.batch_mode, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id,
+                    self.batch_mode,
+                    delivered,
+                    total,
+                    http_failures,
+                    serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(messages, self.batch_mode.content_type(), |mut message| {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = self.payload_to_json(payload)?;
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json)?;
+            serde_json::to_vec(&envelope)
+                .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+        })
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(&self, body: Bytes, count: u64, skipped: u64) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(body, self.batch_mode.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, self.batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, self.batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope = match self.build_envelope(
+                &message,
+                topic_metadata,
+                messages_metadata,
+                payload_json,
+            ) {
+                Ok(env) => env,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (envelope): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");

Review Comment:
   `send_ndjson()` builds a `Vec<String>` (one `String` per message via `serde_json::to_string`), then `lines.join("\n")` allocates a new `String` and copies all the serialized bytes a second time. for a batch of 50 messages at ~1KB each, that's ~50KB of unnecessary copying.
   
   a single pre-sized `Vec<u8>` with `serde_json::to_writer(&mut body, &envelope)` followed by `body.push(b'\n')` avoids both the intermediate `Vec<String>` and the join copy. the per-message error handling / `skipped` counter still works - just track the buffer position before writing and truncate on error.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,

Review Comment:
   also at lines 502, 504, 535, 537, 1027, 1034, 1093, 1116.
   i'm wondering.. will there be a leak of credentials when we print whole url? can someone call have url `https://user@password....`?



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]

Review Comment:
   `#[serde(rename_all = "snake_case")]` forces `NdJson` to serialize as `"nd_json"`, but the industry standard term is `"ndjson"` (one word, per ndjson.org). the `Content-Type` header at line 102 correctly uses `application/x-ndjson`, which highlights the inconsistency.
   
   since this connector is unreleased, now is the cheapest time to fix this - a `#[serde(rename = "ndjson")]` on the `NdJson` variant, or `#[serde(alias = "ndjson")]` to accept both. post-release this becomes a breaking config migration.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)

Review Comment:
   `build_envelope` serializes `MetadataEnvelope` into a `serde_json::Value` here, then callers re-serialize that `Value` to bytes (`to_vec` at line 666, `to_string` at line 750). that's two serialization passes per message.
   
   this exists because `build_envelope` serves three modes - individual needs bytes, json_array needs `Value` for the array, ndjson needs `String`. splitting per-mode would fix it but adds duplication. worth keeping in mind if profiling ever shows serialization as a bottleneck.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986337614


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()
+}
+
+/// Truncate a response body string for log output, respecting UTF-8 char boundaries.
+fn truncate_response(body: &str, max_len: usize) -> &str {
+    if body.len() <= max_len {
+        body
+    } else {
+        // Find the last valid UTF-8 char boundary at or before max_len
+        let end = body.floor_char_boundary(max_len);
+        &body[..end]
+    }
+}
+
+#[async_trait]
+impl Sink for HttpSink {
+    async fn open(&mut self) -> Result<(), Error> {
+        // Validate success_status_codes — empty would cause every response to be treated as failure
+        if self.success_status_codes.is_empty() {
+            return Err(Error::InitError(
+                "success_status_codes must not be empty — would cause retry storms against healthy endpoints".to_string(),
+            ));
+        }
+        for &code in &self.success_status_codes {
+            if !(200..=599).contains(&code) {
+                return Err(Error::InitError(format!(
+                    "Invalid status code {} in success_status_codes — must be 200-599",
+                    code,
+                )));
+            }
+        }
+
+        // Warn if success codes overlap with transient retry codes — these will be treated
+        // as success, silently disabling retry for those status codes.
+        const TRANSIENT_CODES: &[u16] = &[429, 500, 502, 503, 504];
+        let overlap: Vec<u16> = self
+            .success_status_codes
+            .iter()
+            .filter(|c| TRANSIENT_CODES.contains(c))
+            .copied()
+            .collect();
+        if !overlap.is_empty() {
+            warn!(
+                "HTTP sink ID: {} — success_status_codes {:?} overlap with transient retry codes. \
+                 These will be treated as success, disabling retry.",
+                self.id, overlap
+            );
+        }
+
+        // Validate URL
+        if self.url.is_empty() {
+            return Err(Error::InitError(
+                "HTTP sink URL is empty — 'url' is required in [plugin_config]".to_string(),
+            ));
+        }
+        match reqwest::Url::parse(&self.url) {
+            Ok(parsed) => {
+                let scheme = parsed.scheme();
+                if scheme != "http" && scheme != "https" {
+                    return Err(Error::InitError(format!(
+                        "HTTP sink URL scheme '{}' is not allowed — only 'http' and 'https' are supported (url: '{}')",
+                        scheme, self.url,
+                    )));
+                }
+            }
+            Err(e) => {
+                return Err(Error::InitError(format!(
+                    "HTTP sink URL '{}' is not a valid URL: {}",
+                    self.url, e,
+                )));
+            }
+        }
+
+        // Warn if user supplied a Content-Type header — it will be overridden by batch_mode.
+        if self
+            .headers
+            .keys()
+            .any(|k| k.eq_ignore_ascii_case("content-type"))
+        {
+            warn!(
+                "HTTP sink ID: {} — custom 'Content-Type' header in [headers] is ignored. \
+                 Content-Type is set by batch_mode ({:?} -> '{}'). \
+                 Remove it from [headers] to silence this warning.",
+                self.id,
+                self.batch_mode,
+                self.content_type(),
+            );
+        }
+
+        // Validate custom headers — fail fast rather than per-request errors
+        for (key, value) in &self.headers {
+            reqwest::header::HeaderName::from_bytes(key.as_bytes())
+                .map_err(|e| Error::InitError(format!("Invalid header name '{}': {}", key, e)))?;
+            reqwest::header::HeaderValue::from_str(value).map_err(|e| {
+                Error::InitError(format!("Invalid header value for '{}': {}", key, e))
+            })?;
+        }
+
+        // Pre-build the HeaderMap once — avoids re-parsing on every request.
+        // Header names and values were validated above, so expect() is safe here.
+        let mut header_map = reqwest::header::HeaderMap::new();
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue;
+            }
+            let name = reqwest::header::HeaderName::from_bytes(key.as_bytes())
+                .expect("header name validated above");
+            let val = reqwest::header::HeaderValue::from_str(value)
+                .expect("header value validated above");
+            header_map.insert(name, val);
+        }
+        self.request_headers = Some(header_map);
+
+        // Build the HTTP client with config-derived settings
+        self.client = Some(self.build_client()?);
+
+        // Optional health check — uses same pre-built headers and success_status_codes as consume()
+        if self.health_check_enabled {
+            let client = self.client.as_ref().expect("client just built");
+            let headers = self
+                .request_headers
+                .as_ref()
+                .expect("request_headers just built");
+            let health_request =
+                build_request(self.health_check_method, client, &self.url).headers(headers.clone());
+
+            let response = health_request.send().await.map_err(|e| {
+                Error::Connection(format!("Health check failed for URL '{}': {}", self.url, e))
+            })?;
+
+            let status = response.status();
+            if !self.success_status_codes.contains(&status.as_u16()) {
+                return Err(Error::Connection(format!(
+                    "Health check returned status {} (not in success_status_codes {:?}) for URL '{}'",
+                    status.as_u16(),
+                    self.success_status_codes,
+                    self.url,
+                )));
+            }
+
+            info!(
+                "HTTP sink ID: {} — health check passed (status {})",
+                self.id,
+                status.as_u16()
+            );
+        }
+
+        info!(
+            "Opened HTTP sink connector with ID: {} for URL: {} (method: {:?}, \
+             batch_mode: {:?}, timeout: {:?}, max_retries: {})",
+            self.id, self.url, self.method, self.batch_mode, self.timeout, self.max_retries,
+        );
+        Ok(())
+    }
+
+    /// Deliver messages to the configured HTTP endpoint.
+    ///
+    /// **Worst-case latency upper bound** (individual/raw modes):
+    /// `batch_length * (max_retries + 1) * (timeout + max_retry_delay)`.
+    /// Example: 50 * 4 * (30s + 30s) = 12000s. `MAX_CONSECUTIVE_FAILURES` (3)
+    /// mitigates this by aborting early, but a fail-succeed-fail pattern can bypass it.
+    ///
+    /// **Runtime note**: The FFI boundary in `sdk/src/sink.rs` maps `consume()`'s `Result` to
+    /// `i32` (0=ok, 1=err), but the runtime's `process_messages()` in `runtime/src/sink.rs`
+    /// discards that return code. All retry logic lives inside this method — returning `Err`
+    /// does not trigger a runtime-level retry.
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let messages_count = messages.len();
+        if messages_count == 0 {
+            return Ok(());
+        }
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — received {} messages (schema: {}, stream: {}, topic: {})",
+                self.id,
+                messages_count,
+                messages_metadata.schema,
+                topic_metadata.stream,
+                topic_metadata.topic,
+            );
+        }
+
+        let client = self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })?;
+
+        let result = match self.batch_mode {
+            BatchMode::Individual => {
+                self.send_individual(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::Ndjson => {
+                self.send_ndjson(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::JsonArray => {
+                self.send_json_array(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::Raw => self.send_raw(client, messages).await,
+        };
+
+        if let Err(ref e) = result {
+            error!(
+                "HTTP sink ID: {} — consume() returning error (runtime ignores FFI status code): {}",
+                self.id, e
+            );
+        }
+
+        result
+    }
+
+    async fn close(&mut self) -> Result<(), Error> {
+        let requests = self.requests_sent.load(Ordering::Relaxed);
+        let delivered = self.messages_delivered.load(Ordering::Relaxed);
+        let errors = self.errors_count.load(Ordering::Relaxed);
+        let retries = self.retries_count.load(Ordering::Relaxed);
+        let last_success = self.last_success_timestamp.load(Ordering::Relaxed);
+
+        info!(
+            "HTTP sink connector ID: {} closed. Stats: {} requests sent, \
+             {} messages delivered, {} errors, {} retries, last success epoch: {}.",
+            self.id, requests, delivered, errors, retries, last_success,
+        );
+
+        self.request_headers = None;
+        self.client = None;
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use iggy_connector_sdk::Schema;
+
+    // ── Test helpers ──────────────────────────────────────────────────
+
+    /// Parse a JSON string into `simd_json::OwnedValue` for test construction.
+    fn simd_json_from_str(s: &str) -> simd_json::OwnedValue {
+        let mut bytes = s.as_bytes().to_vec();
+        simd_json::to_owned_value(&mut bytes).expect("valid JSON for test")
+    }
+
+    fn given_default_config() -> HttpSinkConfig {
+        HttpSinkConfig {
+            url: "https://api.example.com/ingest".to_string(),
+            method: None,
+            timeout: None,
+            max_payload_size_bytes: None,
+            headers: None,
+            batch_mode: None,
+            include_metadata: None,
+            include_checksum: None,
+            include_origin_timestamp: None,
+            health_check_enabled: None,
+            health_check_method: None,
+            max_retries: None,
+            retry_delay: None,
+            retry_backoff_multiplier: None,
+            max_retry_delay: None,
+            success_status_codes: None,
+            tls_danger_accept_invalid_certs: None,
+            max_connections: None,
+            verbose_logging: None,
+        }
+    }
+
+    fn given_sink_with_defaults() -> HttpSink {
+        HttpSink::new(1, given_default_config())
+    }
+
+    fn given_topic_metadata() -> TopicMetadata {
+        TopicMetadata {
+            stream: "test_stream".to_string(),
+            topic: "test_topic".to_string(),
+        }
+    }
+
+    fn given_messages_metadata() -> MessagesMetadata {
+        MessagesMetadata {
+            partition_id: 0,
+            current_offset: 0,
+            schema: Schema::Json,
+        }
+    }
+
+    fn given_json_message(id: u128, offset: u64) -> ConsumedMessage {
+        ConsumedMessage {
+            id,
+            offset,
+            checksum: 12345,
+            timestamp: 1710064800000000,
+            origin_timestamp: 1710064799000000,
+            headers: None,
+            payload: Payload::Json(simd_json_from_str(r#"{"key":"value"}"#)),
+        }
+    }
+
+    // ── Config resolution tests ──────────────────────────────────────
+
+    #[test]
+    fn given_all_none_config_should_apply_defaults() {
+        let sink = given_sink_with_defaults();
+
+        assert_eq!(sink.method, HttpMethod::Post);
+        assert_eq!(sink.timeout, Duration::from_secs(30));
+        assert_eq!(sink.max_payload_size_bytes, DEFAULT_MAX_PAYLOAD_SIZE);
+        assert_eq!(sink.batch_mode, BatchMode::Individual);
+        assert!(sink.include_metadata);
+        assert!(!sink.include_checksum);
+        assert!(!sink.include_origin_timestamp);
+        assert!(!sink.health_check_enabled);
+        assert_eq!(sink.health_check_method, HttpMethod::Head);
+        assert_eq!(sink.max_retries, DEFAULT_MAX_RETRIES);
+        assert_eq!(sink.retry_delay, Duration::from_secs(1));
+        assert_eq!(sink.retry_backoff_multiplier, DEFAULT_BACKOFF_MULTIPLIER);
+        assert_eq!(sink.max_retry_delay, Duration::from_secs(30));
+        assert_eq!(
+            sink.success_status_codes,
+            HashSet::from([200, 201, 202, 204])
+        );
+        assert!(!sink.tls_danger_accept_invalid_certs);
+        assert_eq!(sink.max_connections, DEFAULT_MAX_CONNECTIONS);
+        assert!(!sink.verbose);
+        assert!(sink.client.is_none());
+    }
+
+    #[test]
+    fn given_explicit_config_values_should_override_defaults() {
+        let config = HttpSinkConfig {
+            url: "https://example.com".to_string(),
+            method: Some(HttpMethod::Put),
+            timeout: Some("10s".to_string()),
+            max_payload_size_bytes: Some(5000),
+            headers: Some(HashMap::from([("X-Key".to_string(), "val".to_string())])),
+            batch_mode: Some(BatchMode::Ndjson),
+            include_metadata: Some(false),
+            include_checksum: Some(true),
+            include_origin_timestamp: Some(true),
+            health_check_enabled: Some(true),
+            health_check_method: Some(HttpMethod::Get),
+            max_retries: Some(5),
+            retry_delay: Some("500ms".to_string()),
+            retry_backoff_multiplier: Some(3.0),
+            max_retry_delay: Some("60s".to_string()),
+            success_status_codes: Some(vec![200, 202]),
+            tls_danger_accept_invalid_certs: Some(true),
+            max_connections: Some(20),
+            verbose_logging: Some(true),
+        };
+
+        let sink = HttpSink::new(1, config);
+        assert_eq!(sink.method, HttpMethod::Put);
+        assert_eq!(sink.timeout, Duration::from_secs(10));
+        assert_eq!(sink.max_payload_size_bytes, 5000);
+        assert_eq!(sink.headers.len(), 1);
+        assert_eq!(sink.batch_mode, BatchMode::Ndjson);
+        assert!(!sink.include_metadata);
+        assert!(sink.include_checksum);
+        assert!(sink.include_origin_timestamp);
+        assert!(sink.health_check_enabled);
+        assert_eq!(sink.health_check_method, HttpMethod::Get);
+        assert_eq!(sink.max_retries, 5);
+        assert_eq!(sink.retry_delay, Duration::from_millis(500));
+        assert_eq!(sink.retry_backoff_multiplier, 3.0);
+        assert_eq!(sink.max_retry_delay, Duration::from_secs(60));
+        assert_eq!(sink.success_status_codes, HashSet::from([200, 202]));
+        assert!(sink.tls_danger_accept_invalid_certs);
+        assert_eq!(sink.max_connections, 20);
+        assert!(sink.verbose);
+    }
+
+    #[test]
+    fn given_backoff_multiplier_below_one_should_clamp_to_one() {
+        let mut config = given_default_config();
+        config.retry_backoff_multiplier = Some(0.5);
+        let sink = HttpSink::new(1, config);
+        assert_eq!(sink.retry_backoff_multiplier, 1.0);
+    }
+
+    #[test]
+    fn given_invalid_duration_string_should_fall_back_to_default() {
+        let mut config = given_default_config();
+        config.timeout = Some("not_a_duration".to_string());
+        config.retry_delay = Some("xyz".to_string());
+        let sink = HttpSink::new(1, config);
+        assert_eq!(sink.timeout, Duration::from_secs(30));
+        assert_eq!(sink.retry_delay, Duration::from_secs(1));
+    }
+
+    // ── Duration parsing tests ───────────────────────────────────────
+
+    #[test]
+    fn given_valid_duration_strings_should_parse_correctly() {
+        let cases = [
+            ("30s", Duration::from_secs(30)),
+            ("500ms", Duration::from_millis(500)),
+            ("2m", Duration::from_secs(120)),
+            ("1h", Duration::from_secs(3600)),
+        ];
+
+        for (input, expected) in cases {
+            assert_eq!(
+                parse_duration(Some(input), "1s"),
+                expected,
+                "input: {}",
+                input
+            );
+        }
+    }
+
+    #[test]
+    fn given_none_duration_should_use_default() {
+        assert_eq!(parse_duration(None, "5s"), Duration::from_secs(5));
+    }
+
+    // ── HttpMethod serde tests ───────────────────────────────────────
+
+    #[test]
+    fn given_http_method_should_serialize_as_uppercase() {
+        let cases = [
+            (HttpMethod::Get, "\"GET\""),
+            (HttpMethod::Head, "\"HEAD\""),
+            (HttpMethod::Post, "\"POST\""),
+            (HttpMethod::Put, "\"PUT\""),
+            (HttpMethod::Patch, "\"PATCH\""),
+            (HttpMethod::Delete, "\"DELETE\""),
+        ];
+
+        for (method, expected_json) in cases {
+            let json = serde_json::to_string(&method).unwrap();
+            assert_eq!(json, expected_json);
+        }
+    }
+
+    #[test]
+    fn given_uppercase_json_should_deserialize_to_method() {
+        let cases = [
+            ("\"GET\"", HttpMethod::Get),
+            ("\"POST\"", HttpMethod::Post),
+            ("\"DELETE\"", HttpMethod::Delete),
+        ];
+
+        for (json, expected) in cases {
+            let method: HttpMethod = serde_json::from_str(json).unwrap();
+            assert_eq!(method, expected);
+        }
+    }
+
+    #[test]
+    fn given_invalid_method_string_should_fail_deserialization() {
+        let result: Result<HttpMethod, _> = serde_json::from_str("\"DELEET\"");
+        assert!(result.is_err());
+    }
+
+    // ── BatchMode serde tests ────────────────────────────────────────
+
+    #[test]
+    fn given_batch_mode_should_serialize_as_snake_case() {
+        let cases = [
+            (BatchMode::Individual, "\"individual\""),
+            (BatchMode::Ndjson, "\"ndjson\""),
+            (BatchMode::JsonArray, "\"json_array\""),
+            (BatchMode::Raw, "\"raw\""),
+        ];
+
+        for (mode, expected_json) in cases {
+            let json = serde_json::to_string(&mode).unwrap();
+            assert_eq!(json, expected_json);
+        }
+    }
+
+    // ── Content-type tests ───────────────────────────────────────────
+
+    #[test]
+    fn given_batch_mode_should_return_correct_content_type() {
+        let cases = [
+            (BatchMode::Individual, "application/json"),
+            (BatchMode::Ndjson, "application/x-ndjson"),
+            (BatchMode::JsonArray, "application/json"),
+            (BatchMode::Raw, "application/octet-stream"),
+        ];
+
+        for (mode, expected) in cases {
+            let mut config = given_default_config();
+            config.batch_mode = Some(mode);
+            let sink = HttpSink::new(1, config);
+            assert_eq!(sink.content_type(), expected);
+        }
+    }
+
+    // ── UUID formatting tests ────────────────────────────────────────
+
+    #[test]
+    fn given_zero_id_should_format_as_valid_v8_uuid() {
+        let result = format_u128_as_uuid(0);
+        let parsed = uuid::Uuid::parse_str(&result).expect("should be valid UUID");
+        assert_eq!(parsed.get_version_num(), 8, "expected v8 UUID");
+        // v8 sets version nibble (byte 6 high) and variant bits (byte 8 high 2)
+        assert_eq!(result, "00000000-0000-8000-8000-000000000000");
+    }
+
+    #[test]
+    fn given_max_u128_should_format_as_valid_v8_uuid() {
+        let result = format_u128_as_uuid(u128::MAX);
+        let parsed = uuid::Uuid::parse_str(&result).expect("should be valid UUID");
+        assert_eq!(parsed.get_version_num(), 8, "expected v8 UUID");
+        assert_eq!(result, "ffffffff-ffff-8fff-bfff-ffffffffffff");
+    }
+
+    #[test]
+    fn given_specific_id_should_produce_valid_v8_uuid() {
+        let id: u128 = 0x0123456789abcdef0123456789abcdef;
+        let result = format_u128_as_uuid(id);
+        let parsed = uuid::Uuid::parse_str(&result).expect("should be valid UUID");
+        assert_eq!(parsed.get_version_num(), 8, "expected v8 UUID");
+        assert_eq!(result.len(), 36, "UUID should be 36 chars");
+        // Original bits preserved except version nibble and variant bits
+        assert_eq!(result, "01234567-89ab-8def-8123-456789abcdef");
+    }
+
+    // ── Truncation tests ─────────────────────────────────────────────
+
+    #[test]
+    fn given_short_string_should_return_unchanged() {
+        assert_eq!(truncate_response("hello", 10), "hello");
+    }
+
+    #[test]
+    fn given_long_string_should_truncate_at_boundary() {
+        let result = truncate_response("hello world", 5);
+        assert_eq!(result, "hello");
+    }
+
+    #[test]
+    fn given_multibyte_string_should_truncate_at_char_boundary() {
+        // "héllo" — 'é' is 2 bytes in UTF-8, so bytes are: h(1) é(2) l(1) l(1) o(1)
+        // floor_char_boundary(2) can't include the 2-byte 'é', returns 1 → "h"
+        let result = truncate_response("héllo", 2);
+        assert_eq!(result, "h");
+    }
+
+    // ── Payload conversion tests ─────────────────────────────────────
+
+    #[test]
+    fn given_json_payload_should_convert_to_serde_json() {
+        let sink = given_sink_with_defaults();
+        let payload = Payload::Json(simd_json_from_str(r#"{"name":"test","count":42}"#));
+
+        let result = sink.payload_to_json(payload).unwrap();
+        assert_eq!(result["name"], "test");
+        assert_eq!(result["count"], 42);
+    }
+
+    #[test]
+    fn given_text_payload_should_convert_to_string_value() {
+        let sink = given_sink_with_defaults();
+        let result = sink
+            .payload_to_json(Payload::Text("hello".to_string()))
+            .unwrap();
+        assert_eq!(result, serde_json::Value::String("hello".to_string()));
+    }
+
+    #[test]
+    fn given_raw_payload_should_base64_encode() {
+        let sink = given_sink_with_defaults();
+        let result = sink.payload_to_json(Payload::Raw(vec![1, 2, 3])).unwrap();
+        assert_eq!(result["iggy_payload_encoding"], "base64");
+        assert_eq!(result["data"], general_purpose::STANDARD.encode([1, 2, 3]));
+    }
+
+    #[test]
+    fn given_flatbuffer_payload_should_base64_encode() {
+        let sink = given_sink_with_defaults();
+        let result = sink
+            .payload_to_json(Payload::FlatBuffer(vec![4, 5, 6]))
+            .unwrap();
+        assert_eq!(result["iggy_payload_encoding"], "base64");
+        assert_eq!(result["data"], general_purpose::STANDARD.encode([4, 5, 6]));
+    }
+
+    #[test]
+    fn given_proto_payload_should_base64_encode_string_bytes() {
+        let sink = given_sink_with_defaults();
+        let result = sink
+            .payload_to_json(Payload::Proto("proto_data".to_string()))
+            .unwrap();
+        assert_eq!(result["iggy_payload_encoding"], "base64");
+        assert_eq!(
+            result["data"],
+            general_purpose::STANDARD.encode(b"proto_data")
+        );
+    }
+
+    // ── Metadata envelope tests ──────────────────────────────────────
+
+    #[test]
+    fn given_include_metadata_true_should_wrap_payload() {
+        let sink = given_sink_with_defaults();
+        let message = given_json_message(42, 10);
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json);
+
+        assert!(envelope.get("metadata").is_some());
+        assert!(envelope.get("payload").is_some());
+
+        let metadata = &envelope["metadata"];
+        assert_eq!(metadata["iggy_offset"], 10);
+        assert_eq!(metadata["iggy_timestamp"], 1710064800000000u64);
+        assert_eq!(metadata["iggy_stream"], "test_stream");
+        assert_eq!(metadata["iggy_topic"], "test_topic");
+        assert_eq!(metadata["iggy_partition_id"], 0);
+        assert_eq!(metadata["iggy_id"], format_u128_as_uuid(42));
+        // Verify conditional fields are absent by default
+        assert!(metadata.get("iggy_checksum").is_none());
+        assert!(metadata.get("iggy_origin_timestamp").is_none());
+    }
+
+    #[test]
+    fn given_include_metadata_false_should_return_raw_payload() {
+        let mut config = given_default_config();
+        config.include_metadata = Some(false);
+        let sink = HttpSink::new(1, config);
+
+        let message = given_json_message(1, 0);
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json.clone());
+
+        // Should be the payload itself, not wrapped
+        assert_eq!(envelope, payload_json);
+        assert!(envelope.get("metadata").is_none());
+    }
+
+    #[test]
+    fn given_include_checksum_should_add_checksum_to_metadata() {
+        let mut config = given_default_config();
+        config.include_checksum = Some(true);
+        let sink = HttpSink::new(1, config);
+
+        let message = given_json_message(1, 0);
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json);
+        assert_eq!(envelope["metadata"]["iggy_checksum"], 12345);
+    }
+
+    #[test]
+    fn given_include_origin_timestamp_should_add_to_metadata() {
+        let mut config = given_default_config();
+        config.include_origin_timestamp = Some(true);
+        let sink = HttpSink::new(1, config);
+
+        let message = given_json_message(1, 0);
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json);
+        assert_eq!(
+            envelope["metadata"]["iggy_origin_timestamp"],
+            1710064799000000u64
+        );
+    }
+
+    #[test]
+    fn given_message_with_headers_should_include_iggy_headers_in_metadata() {
+        use iggy_connector_sdk::ConsumedMessage;
+
+        let sink = given_sink_with_defaults();
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+
+        let mut headers = HashMap::new();
+        headers.insert(
+            "x-correlation-id".parse().unwrap(),
+            "abc-123".parse().unwrap(),
+        );
+
+        let message = ConsumedMessage {
+            id: 1,
+            offset: 0,
+            checksum: 0,
+            timestamp: 1710064800000000,
+            origin_timestamp: 0,
+            headers: Some(headers),
+            payload: Payload::Json(simd_json_from_str(r#"{"key":"value"}"#)),
+        };
+
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json);
+
+        let iggy_headers = &envelope["metadata"]["iggy_headers"];
+        assert!(
+            !iggy_headers.is_null(),
+            "Expected iggy_headers in metadata when message has headers"
+        );
+        assert!(
+            iggy_headers.get("x-correlation-id").is_some(),
+            "Expected header key in iggy_headers, got: {iggy_headers}"
+        );
+    }
+
+    #[test]
+    fn given_message_without_headers_should_not_include_iggy_headers() {
+        let sink = given_sink_with_defaults();
+        let message = given_json_message(1, 0);
+        let topic_meta = given_topic_metadata();
+        let msg_meta = given_messages_metadata();
+        let payload_json = sink.payload_to_json(message.payload.clone()).unwrap();
+
+        let envelope = sink.build_envelope(&message, &topic_meta, &msg_meta, payload_json);
+        assert!(
+            envelope["metadata"].get("iggy_headers").is_none(),

Review Comment:
   For example, it's one of the places where having the const for particular field name, or even better, the strongly typed struct, would make such a test less error-prone :).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986301141


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({

Review Comment:
   Also here - IMHO better to use the strongly typed struct and then just serialize as json as a result.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986304830


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,

Review Comment:
   Why not to use `reqwest-middleware` which is already part of the workspace, connectors runtime etc.?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio merged PR #2925:
URL: https://github.com/apache/iggy/pull/2925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986324075


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()

Review Comment:
   Actually, do we need `v8`, isn't `v4` sufficient enough? Maybe let's return the `simple` variant (without the dashes) or let the user configure it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2957075625


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {

Review Comment:
   ok, will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4062894483

   **[critical] `payload.clone()` on every message in all 4 batch mode loops**
   `lib.rs:572, 727, 799, 880`
   
   `consume()` takes `messages: Vec<ConsumedMessage>` by value, but all send methods iterate via `&messages` and `.clone()` each payload. for `Payload::Raw(Vec<u8>)`, `FlatBuffer(Vec<u8>)`, and `Json(OwnedValue)` this is a full heap copy per message. fix is trivial -- change to `for message in messages` (consuming iterator) since `payload_to_json()` and `try_into_vec()` already take `Payload` by value.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2990449401


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()

Review Comment:
   @spetz Not to overcomplicate this, but I forgot to include an alternative consideration for completeness. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986286087


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {

Review Comment:
   Maybe the `content_type` could be a function exposed directly in `BatchMode`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4130361870

   @spetz and @hubcio 
   
   Please note the limitations that were highlighted due to shift to `reqwest-middleware` . It is fine as it is, but something for you consideration. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r3014695852


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,
+                body.len(),
+            );
+        }
+
+        self.send_attempts.fetch_add(1, Ordering::Relaxed);
+
+        let response = build_request(self.method, client, &self.url)
+            .headers(headers.clone())
+            .header("content-type", content_type)
+            .body(body)
+            .send()
+            .await
+            .map_err(|e| {
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                error!(
+                    "HTTP sink ID: {} — request to {} failed after middleware retries: {:#}",
+                    self.id, self.url, e
+                );
+                Error::HttpRequestFailed(format!("HTTP {} — {}", self.url, e))
+            })?;
+
+        let status = response.status();
+        // success_status_codes is checked in BOTH the retry strategy (to stop retrying)
+        // AND here (to classify the final response). Both must use the same set.
+        if self.success_status_codes.contains(&status.as_u16()) {
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — success (status {})",
+                    self.id,
+                    status.as_u16()
+                );
+            }
+            self.record_success();
+            return Ok(());
+        }
+
+        // Non-success status after middleware exhausted retries — read body for diagnostics
+        let response_body = match response.text().await {
+            Ok(body) => body,
+            Err(e) => format!("<body read error: {}>", e),
+        };
+
+        error!(
+            "HTTP sink ID: {} — request failed (status {}). Response: {}",
+            self.id,
+            status.as_u16(),
+            truncate_response(&response_body, 500),
+        );
+        self.errors_count.fetch_add(1, Ordering::Relaxed);
+        Err(Error::HttpRequestFailed(format!(
+            "HTTP {} — status: {}",
+            self.url,
+            status.as_u16()
+        )))
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    self.batch_mode,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self.send_with_retry(Bytes::from(body), content_type).await {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, self.batch_mode, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id,
+                    self.batch_mode,
+                    delivered,
+                    total,
+                    http_failures,
+                    serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(messages, self.batch_mode.content_type(), |mut message| {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = self.payload_to_json(payload)?;
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json)?;
+            serde_json::to_vec(&envelope)
+                .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+        })
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(&self, body: Bytes, count: u64, skipped: u64) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(body, self.batch_mode.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, self.batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, self.batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope = match self.build_envelope(
+                &message,
+                topic_metadata,
+                messages_metadata,
+                payload_json,
+            ) {
+                Ok(env) => env,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (envelope): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");

Review Comment:
   `send_ndjson()` builds a `Vec<String>` (one `String` per message via `serde_json::to_string`), then `lines.join("\n")` allocates a new `String` and copies all the serialized bytes a second time. for a batch of 50 messages at ~1KB each, that's ~50KB of unnecessary copying.
   
   a single pre-sized `Vec<u8>` with `serde_json::to_writer(&mut body, &envelope)` followed by `body.push(b'\n')` avoids both the intermediate `Vec<String>` and the join copy. the per-message error handling / `skipped` counter still works - just track the buffer position before writing and truncate on error.
   
   also, instead of callign `Vec::new()` use `Vec::with_capacity()` to avoid realloc (malloc + memcpy underneath, or memmove between pages)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2936866483


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let request = self
+                .request_builder(client)
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    /// Continues processing remaining messages if one fails (partial delivery).
+    /// Aborts remaining messages after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures
+    /// to avoid hammering a dead endpoint.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            let body = match serde_json::to_vec(&envelope) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to serialize envelope at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Envelope serialize: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver message at offset {} after retries: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for message in &messages {
+            let payload_json = match self.payload_to_json(message.payload.clone()) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    /// Only meaningful for Raw/FlatBuffer/Proto payloads; JSON/Text are sent as UTF-8 bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in &messages {
+            let offset = message.offset;
+            let body = match message.payload.clone().try_into_vec() {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to convert raw payload at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(Error::Serialization(format!("Raw payload convert: {}", e)));
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — raw payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Raw payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), self.content_type())
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver raw message at offset {}: {}",
+                        self.id, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting raw batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial raw delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+}
+
+/// Convert `simd_json::OwnedValue` to `serde_json::Value` via direct structural mapping.
+/// NaN/Infinity f64 values are mapped to `null` (same as Elasticsearch sink).
+fn owned_value_to_serde_json(value: &simd_json::OwnedValue) -> serde_json::Value {

Review Comment:
   I tried to maintain isolation away from changes to anything other than connectors. Am I correct to assume that you would like to see this in sdk?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4063188570

   > overall good direction, please fix the CI (markdown lint) and rest of the comments. looks pretty solid to me.
   
   Thank you for the notes and valuable insight. For certain, I will fix and commit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r3014695840


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,

Review Comment:
   also at lines 502, 504, 535, 537, 1027, 1034, 1093, 1116.
   i'm wondering.. will there be a leak of credentials when we print whole url? can someone call have url `https://user:password....`?



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,

Review Comment:
   also at lines 502, 504, 535, 537, 1027, 1034, 1093, 1116.
   i'm wondering.. will there be a leak of credentials when we print whole url? can someone call have url `https://user:password@domain....`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986340838


##########
core/connectors/sinks/http_sink/config.toml:
##########
@@ -0,0 +1,90 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+type = "sink"
+key = "http"
+enabled = true
+version = 0
+name = "HTTP sink"
+path = "../../target/release/libiggy_connector_http_sink"
+verbose = false
+
+[[streams]]
+stream = "my_stream"
+topics = ["my_topic"]
+schema = "json"
+batch_length = 50
+poll_interval = "100ms"
+consumer_group = "http_sink_group"
+
+[plugin_config]
+# Required — target URL for HTTP requests.
+url = "https://api.example.com/ingest"
+
+# HTTP method (default: POST). Valid: GET, HEAD, POST, PUT, PATCH, DELETE.
+method = "POST"
+
+# Request timeout (default: 30s).
+timeout = "30s"
+
+# Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+max_payload_size_bytes = 10485760
+
+# Payload formatting mode (default: individual).
+# - "individual": one HTTP request per message
+# - "ndjson": newline-delimited JSON, all messages in one request
+# - "json_array": JSON array of messages in one request
+# - "raw": raw bytes, individual requests only
+batch_mode = "ndjson"

Review Comment:
   If the enum would be changed to `NdJson` please remember to update the configs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4103541017

   Good catch — I wasn't using `prek`. Installed it now (`cargo install prek && prek install`) and ran the hooks. The `cargo fmt` diff has been committed and pushed in `3cae7ad1`. CI should be green on the latest push.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986278048


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,

Review Comment:
   Should it be `NdJson` (`nd_json`) instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986333305


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()
+}
+
+/// Truncate a response body string for log output, respecting UTF-8 char boundaries.
+fn truncate_response(body: &str, max_len: usize) -> &str {
+    if body.len() <= max_len {
+        body
+    } else {
+        // Find the last valid UTF-8 char boundary at or before max_len
+        let end = body.floor_char_boundary(max_len);
+        &body[..end]
+    }
+}
+
+#[async_trait]
+impl Sink for HttpSink {
+    async fn open(&mut self) -> Result<(), Error> {
+        // Validate success_status_codes — empty would cause every response to be treated as failure
+        if self.success_status_codes.is_empty() {
+            return Err(Error::InitError(
+                "success_status_codes must not be empty — would cause retry storms against healthy endpoints".to_string(),
+            ));
+        }
+        for &code in &self.success_status_codes {
+            if !(200..=599).contains(&code) {
+                return Err(Error::InitError(format!(
+                    "Invalid status code {} in success_status_codes — must be 200-599",
+                    code,
+                )));
+            }
+        }
+
+        // Warn if success codes overlap with transient retry codes — these will be treated
+        // as success, silently disabling retry for those status codes.
+        const TRANSIENT_CODES: &[u16] = &[429, 500, 502, 503, 504];
+        let overlap: Vec<u16> = self
+            .success_status_codes
+            .iter()
+            .filter(|c| TRANSIENT_CODES.contains(c))
+            .copied()
+            .collect();
+        if !overlap.is_empty() {
+            warn!(
+                "HTTP sink ID: {} — success_status_codes {:?} overlap with transient retry codes. \
+                 These will be treated as success, disabling retry.",
+                self.id, overlap
+            );
+        }
+
+        // Validate URL
+        if self.url.is_empty() {
+            return Err(Error::InitError(
+                "HTTP sink URL is empty — 'url' is required in [plugin_config]".to_string(),
+            ));
+        }
+        match reqwest::Url::parse(&self.url) {
+            Ok(parsed) => {
+                let scheme = parsed.scheme();
+                if scheme != "http" && scheme != "https" {
+                    return Err(Error::InitError(format!(
+                        "HTTP sink URL scheme '{}' is not allowed — only 'http' and 'https' are supported (url: '{}')",
+                        scheme, self.url,
+                    )));
+                }
+            }
+            Err(e) => {
+                return Err(Error::InitError(format!(
+                    "HTTP sink URL '{}' is not a valid URL: {}",
+                    self.url, e,
+                )));
+            }
+        }
+
+        // Warn if user supplied a Content-Type header — it will be overridden by batch_mode.
+        if self
+            .headers
+            .keys()
+            .any(|k| k.eq_ignore_ascii_case("content-type"))
+        {
+            warn!(
+                "HTTP sink ID: {} — custom 'Content-Type' header in [headers] is ignored. \
+                 Content-Type is set by batch_mode ({:?} -> '{}'). \
+                 Remove it from [headers] to silence this warning.",
+                self.id,
+                self.batch_mode,
+                self.content_type(),
+            );
+        }
+
+        // Validate custom headers — fail fast rather than per-request errors
+        for (key, value) in &self.headers {
+            reqwest::header::HeaderName::from_bytes(key.as_bytes())
+                .map_err(|e| Error::InitError(format!("Invalid header name '{}': {}", key, e)))?;
+            reqwest::header::HeaderValue::from_str(value).map_err(|e| {
+                Error::InitError(format!("Invalid header value for '{}': {}", key, e))
+            })?;
+        }
+
+        // Pre-build the HeaderMap once — avoids re-parsing on every request.
+        // Header names and values were validated above, so expect() is safe here.
+        let mut header_map = reqwest::header::HeaderMap::new();
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue;
+            }
+            let name = reqwest::header::HeaderName::from_bytes(key.as_bytes())
+                .expect("header name validated above");
+            let val = reqwest::header::HeaderValue::from_str(value)
+                .expect("header value validated above");
+            header_map.insert(name, val);
+        }
+        self.request_headers = Some(header_map);
+
+        // Build the HTTP client with config-derived settings
+        self.client = Some(self.build_client()?);
+
+        // Optional health check — uses same pre-built headers and success_status_codes as consume()
+        if self.health_check_enabled {
+            let client = self.client.as_ref().expect("client just built");
+            let headers = self
+                .request_headers
+                .as_ref()
+                .expect("request_headers just built");
+            let health_request =
+                build_request(self.health_check_method, client, &self.url).headers(headers.clone());
+
+            let response = health_request.send().await.map_err(|e| {
+                Error::Connection(format!("Health check failed for URL '{}': {}", self.url, e))
+            })?;
+
+            let status = response.status();
+            if !self.success_status_codes.contains(&status.as_u16()) {
+                return Err(Error::Connection(format!(
+                    "Health check returned status {} (not in success_status_codes {:?}) for URL '{}'",
+                    status.as_u16(),
+                    self.success_status_codes,
+                    self.url,
+                )));
+            }
+
+            info!(
+                "HTTP sink ID: {} — health check passed (status {})",
+                self.id,
+                status.as_u16()
+            );
+        }
+
+        info!(
+            "Opened HTTP sink connector with ID: {} for URL: {} (method: {:?}, \
+             batch_mode: {:?}, timeout: {:?}, max_retries: {})",
+            self.id, self.url, self.method, self.batch_mode, self.timeout, self.max_retries,
+        );
+        Ok(())
+    }
+
+    /// Deliver messages to the configured HTTP endpoint.
+    ///
+    /// **Worst-case latency upper bound** (individual/raw modes):
+    /// `batch_length * (max_retries + 1) * (timeout + max_retry_delay)`.
+    /// Example: 50 * 4 * (30s + 30s) = 12000s. `MAX_CONSECUTIVE_FAILURES` (3)
+    /// mitigates this by aborting early, but a fail-succeed-fail pattern can bypass it.
+    ///
+    /// **Runtime note**: The FFI boundary in `sdk/src/sink.rs` maps `consume()`'s `Result` to
+    /// `i32` (0=ok, 1=err), but the runtime's `process_messages()` in `runtime/src/sink.rs`
+    /// discards that return code. All retry logic lives inside this method — returning `Err`
+    /// does not trigger a runtime-level retry.
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let messages_count = messages.len();
+        if messages_count == 0 {
+            return Ok(());
+        }
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — received {} messages (schema: {}, stream: {}, topic: {})",
+                self.id,
+                messages_count,
+                messages_metadata.schema,
+                topic_metadata.stream,
+                topic_metadata.topic,
+            );
+        }
+
+        let client = self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })?;
+
+        let result = match self.batch_mode {
+            BatchMode::Individual => {
+                self.send_individual(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::Ndjson => {
+                self.send_ndjson(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::JsonArray => {
+                self.send_json_array(client, topic_metadata, &messages_metadata, messages)
+                    .await
+            }
+            BatchMode::Raw => self.send_raw(client, messages).await,
+        };
+
+        if let Err(ref e) = result {
+            error!(
+                "HTTP sink ID: {} — consume() returning error (runtime ignores FFI status code): {}",
+                self.id, e
+            );
+        }
+
+        result
+    }
+
+    async fn close(&mut self) -> Result<(), Error> {
+        let requests = self.requests_sent.load(Ordering::Relaxed);
+        let delivered = self.messages_delivered.load(Ordering::Relaxed);
+        let errors = self.errors_count.load(Ordering::Relaxed);
+        let retries = self.retries_count.load(Ordering::Relaxed);
+        let last_success = self.last_success_timestamp.load(Ordering::Relaxed);
+
+        info!(
+            "HTTP sink connector ID: {} closed. Stats: {} requests sent, \
+             {} messages delivered, {} errors, {} retries, last success epoch: {}.",
+            self.id, requests, delivered, errors, retries, last_success,
+        );
+
+        self.request_headers = None;
+        self.client = None;
+        Ok(())
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use iggy_connector_sdk::Schema;
+
+    // ── Test helpers ──────────────────────────────────────────────────

Review Comment:
   Let's remove comment sections like this and the other ones below.
   Also, code should be read like a book when possible, meaning that any helper functions etc. could be placed below the ones actually invoking them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2990387109


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()

Review Comment:
   The server's code uses v4 as the default, and since HTTP passes ID as a string, I think that the simple from (without dashes) would be sufficient.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4152546275

   @mlevkov looks like testcase you added is failing in the CI: 
   <img width="1878" height="758" alt="image" src="https://github.com/user-attachments/assets/a921dca1-e973-46b7-882b-f6cc09a6d9ae" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4119679077

   > looking very good, I found only one nit: you moved `owned_value_to_serde_json` to connectors SDK but you didn't use it in `elasticsearch_sink`. there is duplicate code. please use it :)
   > 
   > after you fix that we can merge. also, please fix CI - there were some changes in the `binary_protocol` crate, looks like tests fail after rebase.
   
   @hubcio Originally, I did, but since I was not a maintainer of Elasticsearch connector I chose not to touch it, however, now that you mention I will do so. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986292303


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),

Review Comment:
   Let's avoid using `serde_json::json!` whenever possible, and simply pass the strongly typed struct instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986289774


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"

Review Comment:
   I think that both fields `data` and `iggy_payload_encoding` should be added as e.g. `pub const` so it's easy to use them in multiple places (and test)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["codecov[bot] (via GitHub)" <gi...@apache.org>]

codecov[bot] commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4052992590

   ## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/2925?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 64.87%. Comparing base ([`261d255`](https://app.codecov.io/gh/apache/iggy/commit/261d25557d9aabdd51cb071ed8f62422ccb70c90?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ec179a9`](https://app.codecov.io/gh/apache/iggy/commit/ec179a9e5f6c6ad41a37037902ec0f3a99f4dbb8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2925      +/-   ##
   ============================================
   - Coverage     70.08%   64.87%   -5.22%     
     Complexity      776      776              
   ============================================
     Files          1028     1028              
     Lines         85279    85279              
     Branches      62653    62663      +10     
   ============================================
   - Hits          59771    55327    -4444     
   - Misses        22980    27727    +4747     
   + Partials       2528     2225     -303     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/iggy/pull/2925/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [csharp](https://app.codecov.io/gh/apache/iggy/pull/2925/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `12.56% <ø> (-55.06%)` | :arrow_down: |
   | [go](https://app.codecov.io/gh/apache/iggy/pull/2925/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.37% <ø> (-32.00%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   [see 124 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/2925/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2936683260


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2099 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::HashMap;
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: Vec<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204]);
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Apply the configured HTTP method to a `reqwest::Client` for the target URL,
+    /// including custom headers (excluding Content-Type, which is set per-request by batch mode).
+    fn request_builder(&self, client: &reqwest::Client) -> reqwest::RequestBuilder {
+        let mut builder = build_request(self.method, client, &self.url);
+        for (key, value) in &self.headers {
+            if key.eq_ignore_ascii_case("content-type") {
+                continue; // Content-Type is set by batch mode in send_with_retry
+            }
+            builder = builder.header(key, value);
+        }
+        builder
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,

Review Comment:
   `iggy_timestamp` (server-assigned) vs `iggy_origin_timestamp` (producer-supplied) -- the difference isn't obvious from the names. these are new public api surface. once shipped, the names become a contract. consider `iggy_server_timestamp` or at least document the distinction prominently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986308587


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,

Review Comment:
   Let's use the basic enum with e.g. `strum` macros on top to display the mode properly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2990343359


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);
+        }
+
+        if self.include_origin_timestamp {
+            metadata["iggy_origin_timestamp"] = serde_json::json!(message.origin_timestamp);
+        }
+
+        if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let headers_map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        serde_json::json!({
+                            "data": general_purpose::STANDARD.encode(raw),
+                            "iggy_header_encoding": "base64"
+                        })
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    (k.to_string_value(), value)
+                })
+                .collect();
+            metadata["iggy_headers"] = serde_json::Value::Object(headers_map);
+        }
+
+        serde_json::json!({
+            "metadata": metadata,
+            "payload": payload_json,
+        })
+    }
+
+    /// Classify whether an HTTP status code is transient (worth retrying).
+    fn is_transient_status(status: reqwest::StatusCode) -> bool {
+        matches!(status.as_u16(), 429 | 500 | 502 | 503 | 504)
+    }
+
+    /// Extract `Retry-After` header value as a Duration (seconds), capped to `max_retry_delay`.
+    fn parse_retry_after(&self, response: &reqwest::Response) -> Option<Duration> {
+        let header_raw = response.headers().get(reqwest::header::RETRY_AFTER)?;
+        let header_value = match header_raw.to_str() {
+            Ok(s) => s,
+            Err(e) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header contains non-ASCII bytes: {}. \
+                     Using computed backoff.",
+                    self.id, e,
+                );
+                return None;
+            }
+        };
+        match header_value.parse::<u64>() {
+            Ok(secs) => Some(Duration::from_secs(secs).min(self.max_retry_delay)),
+            Err(_) => {
+                warn!(
+                    "HTTP sink ID: {} — Retry-After header '{}' is not an integer delay; \
+                     HTTP-date format is not supported. Using computed backoff.",
+                    self.id, header_value,
+                );
+                None
+            }
+        }
+    }
+
+    /// Compute the retry delay for a given attempt, applying exponential backoff
+    /// capped at `max_retry_delay`. Clamps before `Duration::from_secs_f64` to avoid
+    /// panics when extreme backoff configs produce infinity (e.g., multiplier=1000, retries=200).
+    fn compute_retry_delay(&self, attempt: u32) -> Duration {
+        let delay_secs =
+            self.retry_delay.as_secs_f64() * self.retry_backoff_multiplier.powi(attempt as i32);
+        let capped_secs = delay_secs.min(self.max_retry_delay.as_secs_f64());
+        if !capped_secs.is_finite() || capped_secs < 0.0 {
+            return self.max_retry_delay;
+        }
+        Duration::from_secs_f64(capped_secs)
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry logic. Returns Ok on success, Err after exhausting retries.
+    ///
+    /// Takes `Bytes` instead of `Vec<u8>` so retries clone via reference-count increment (O(1))
+    /// rather than copying the entire payload on each attempt.
+    async fn send_with_retry(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        content_type: &str,
+    ) -> Result<(), Error> {
+        let mut attempt = 0u32;
+
+        loop {
+            let headers = self.request_headers.as_ref().ok_or_else(|| {
+                Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+            })?;
+            let request = build_request(self.method, client, &self.url)
+                .headers(headers.clone())
+                .header("content-type", content_type)
+                .body(body.clone())
+                .build()
+                .map_err(|e| Error::HttpRequestFailed(format!("Request build error: {}", e)))?;
+
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — sending {} {} (attempt {}/{}, {} bytes)",
+                    self.id,
+                    request.method(),
+                    request.url(),
+                    attempt + 1,
+                    self.max_retries + 1,
+                    body.len(),
+                );
+            }
+
+            self.requests_sent.fetch_add(1, Ordering::Relaxed);
+
+            match client.execute(request).await {
+                Ok(response) => {
+                    let status = response.status();
+
+                    // Check for Retry-After before consuming the response
+                    let retry_after = self.parse_retry_after(&response);
+
+                    if self.success_status_codes.contains(&status.as_u16()) {
+                        if self.verbose {
+                            debug!(
+                                "HTTP sink ID: {} — success (status {})",
+                                self.id,
+                                status.as_u16()
+                            );
+                        }
+                        self.record_success();
+                        return Ok(());
+                    }
+
+                    // Non-success status — read body for diagnostics
+                    let response_body = match response.text().await {
+                        Ok(body) => body,
+                        Err(e) => format!("<body read error: {}>", e),
+                    };
+
+                    if Self::is_transient_status(status) && attempt < self.max_retries {
+                        let delay =
+                            retry_after.unwrap_or_else(|| self.compute_retry_delay(attempt));
+                        warn!(
+                            "HTTP sink ID: {} — transient error (status {}, attempt {}/{}). \
+                             Retrying in {:?}. Response: {}",
+                            self.id,
+                            status.as_u16(),
+                            attempt + 1,
+                            self.max_retries + 1,
+                            delay,
+                            truncate_response(&response_body, 200),
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    // Non-transient or retries exhausted
+                    error!(
+                        "HTTP sink ID: {} — request failed (status {}, attempt {}/{}). \
+                         Response: {}",
+                        self.id,
+                        status.as_u16(),
+                        attempt + 1,
+                        self.max_retries + 1,
+                        truncate_response(&response_body, 500),
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "HTTP {} — status: {}",
+                        self.url,
+                        status.as_u16()
+                    )));
+                }
+                Err(network_err) => {
+                    if attempt < self.max_retries {
+                        let delay = self.compute_retry_delay(attempt);
+                        warn!(
+                            "HTTP sink ID: {} — network error (attempt {}/{}): {}. \
+                             Retrying in {:?}.",
+                            self.id,
+                            attempt + 1,
+                            self.max_retries + 1,
+                            network_err,
+                            delay,
+                        );
+                        self.retries_count.fetch_add(1, Ordering::Relaxed);
+                        tokio::time::sleep(delay).await;
+                        attempt += 1;
+                        continue;
+                    }
+
+                    error!(
+                        "HTTP sink ID: {} — network error after {} attempts: {}",
+                        self.id,
+                        attempt + 1,
+                        network_err,
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    return Err(Error::HttpRequestFailed(format!(
+                        "Network error after {} attempts: {}",
+                        attempt + 1,
+                        network_err
+                    )));
+                }
+            }
+        }
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mode_name: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    mode_name,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self
+                .send_with_retry(client, Bytes::from(body), content_type)
+                .await
+            {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, mode_name, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, mode_name, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id, mode_name, delivered, total, http_failures, serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(
+            client,
+            messages,
+            self.content_type(),
+            "individual",
+            |mut message| {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+                let payload_json = self.payload_to_json(payload)?;
+                let envelope =
+                    self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+                serde_json::to_vec(&envelope)
+                    .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+            },
+        )
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(
+        &self,
+        client: &reqwest::Client,
+        body: Bytes,
+        count: u64,
+        skipped: u64,
+        batch_mode: &str,
+    ) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(client, body, self.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");
+        body_str.push('\n'); // NDJSON spec requires trailing newline
+        let body = body_str.into_bytes();
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — NDJSON batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "NDJSON batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "NDJSON")
+            .await
+    }
+
+    /// Send messages in `json_array` mode — all messages as a single JSON array.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_json_array(
+        &self,
+        client: &reqwest::Client,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut envelopes = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in JSON array batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json);
+            envelopes.push(envelope);
+        }
+
+        if envelopes.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in JSON array batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = envelopes.len() as u64;
+
+        let body = match serde_json::to_vec(&envelopes) {
+            Ok(b) => b,
+            Err(e) => {
+                error!(
+                    "HTTP sink ID: {} — failed to serialize JSON array batch \
+                     ({} envelopes, {} skipped): {}",
+                    self.id,
+                    envelopes.len(),
+                    skipped,
+                    e,
+                );
+                // Count all successfully-built envelopes as errors (skipped already counted individually)
+                self.errors_count.fetch_add(count, Ordering::Relaxed);
+                return Err(Error::Serialization(format!(
+                    "JSON array serialize ({} envelopes): {}",
+                    envelopes.len(),
+                    e
+                )));
+            }
+        };
+
+        if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+            error!(
+                "HTTP sink ID: {} — JSON array batch exceeds max payload size ({} > {} bytes)",
+                self.id,
+                body.len(),
+                self.max_payload_size_bytes,
+            );
+            // Count all successfully-serialized messages as errors (skipped already counted individually)
+            self.errors_count.fetch_add(count, Ordering::Relaxed);
+            return Err(Error::HttpRequestFailed(format!(
+                "JSON array batch exceeds max size: {} bytes",
+                body.len()
+            )));
+        }
+
+        self.send_batch_body(client, Bytes::from(body), count, skipped, "JSON array")
+            .await
+    }
+
+    /// Send messages in `raw` mode — one HTTP request per message with raw bytes.
+    async fn send_raw(
+        &self,
+        client: &reqwest::Client,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(client, messages, self.content_type(), "raw", |message| {
+            message
+                .payload
+                .try_into_vec()
+                .map_err(|e| Error::Serialization(format!("Raw payload convert: {}", e)))
+        })
+        .await
+    }
+}
+
+/// Map an `HttpMethod` to a `reqwest::RequestBuilder` for the given URL.
+fn build_request(
+    method: HttpMethod,
+    client: &reqwest::Client,
+    url: &str,
+) -> reqwest::RequestBuilder {
+    match method {
+        HttpMethod::Get => client.get(url),
+        HttpMethod::Head => client.head(url),
+        HttpMethod::Post => client.post(url),
+        HttpMethod::Put => client.put(url),
+        HttpMethod::Patch => client.patch(url),
+        HttpMethod::Delete => client.delete(url),
+    }
+}
+
+/// Format a u128 message ID as an RFC 4122 v8 (custom) UUID.
+///
+/// Uses `Uuid::new_v8()` which sets version=8 and variant=RFC4122 bits,
+/// producing UUIDs that downstream libraries accept as valid.
+/// Note: `new_v8()` overwrites 6 bits (version nibble + variant bits), so the
+/// UUID is not round-trippable to the original u128 value.
+fn format_u128_as_uuid(id: u128) -> String {
+    uuid::Uuid::new_v8(id.to_be_bytes()).to_string()

Review Comment:
   Great question. Here's the trade-off analysis:
   
   ### UUID format options
   
   | Option | What happens | Pros | Cons |
   |--------|-------------|------|------|
   | **v4 (random)** | Fresh random UUID per message | Valid RFC 4122, universally accepted | Loses all correlation to Iggy message ID — can't trace back to source message |
   | **v8 (current)** | Encodes u128 message ID with version/variant bits | Valid UUID, downstream libraries accept it, mostly preserves original ID | Overwrites 6 bits — not round-trippable, theoretical 1-in-64 collision window |
   | **Simple hex** | `01234567-89ab-cdef-0123-456789abcdef` (no version bits) | Lossless — exact u128 preserved, fully round-trippable | Not a valid UUID — downstream `parse_uuid()` may reject it |
   | **Configurable** | User picks format in config | Maximum flexibility | Adds config surface, complexity, testing burden for a display format |
   
   ### Recommendation
   
   Keep **simple hex** (lossless, original approach) — it preserves the exact Iggy message ID for debugging/correlation, which is more valuable for an Iggy connector than strict UUID compliance. Downstream consumers already treat `iggy_id` as an opaque identifier, not as a UUID they need to validate.
   
   If strict UUID compliance is needed, v8 is the right choice — but it sacrifices 6 bits of the original ID.
   
   Happy to go with whatever you prefer here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4155257194

   I will take a look, thank you for a note.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4102974688

   are you using `prek` precommit hooks? see contributing.md in the root of the repository, this way you won't be able to push broken code to PR :) i noticed that because clippy / rustfmt has failed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4128950191

   Thank you @spetz — excellent feedback across the board. Here's how I've grouped the 12 comments for implementation:
   
   ### Grouped by theme
   
   **A. Constants & naming** (straightforward)
   | # | Line | Change |
   |---|------|--------|
   | 1 | 77 | `Ndjson` → `NdJson` (`nd_json` in snake_case) |
   | 3 | 318 | `pub const` for `"data"`, `"iggy_payload_encoding"` field names |
   | 5 | 349 | Constants for all metadata field names (`iggy_checksum`, `iggy_origin_timestamp`, etc.) |
   | 11 | 1184 | Remove `// ── ... ──` comment sections; reorder helpers below callers |
   | 12 | 1669 | Use constants for field names in test assertions |
   
   **B. Strongly typed structs + enum methods**
   | # | Line | Change |
   |---|------|--------|
   | 2 | 295 | Move `content_type()` to `BatchMode` impl |
   | 4/6 | 340/378 | Replace `serde_json::json!` with serializable `MetadataEnvelope` struct |
   | 8 | 584 | `strum` Display on `BatchMode` for `mode_name` instead of string parameter |
   
   **C. Method signature cleanup**
   | # | Line | Change |
   |---|------|--------|
   | 9 | 444 | Use `self.client` directly instead of passing `client` parameter |
   
   **D. Design decisions (want to discuss)**
   | # | Line | Question |
   |---|------|----------|
   | 7 | 443 | `reqwest-middleware` — would replace hand-rolled retry with the workspace's middleware + retry policy. Leaning toward yes — aligns with the rest of the connectors runtime and gets retry improvements for free on workspace upgrades. |
   | 10 | 942 | UUID format — currently v8 (per hubcio's earlier request for proper RFC 4122 bits). Options: (a) v4 random UUID (loses correlation to iggy message ID), (b) v8 custom (current — preserves most of the ID but overwrites 6 bits), (c) simple hex without dashes (lossless but not a valid UUID), (d) user-configurable. What's your preference? |
   
   Will work through A → B → C → D in order. Addressing #10 and #7 last since they need alignment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["hubcio (via GitHub)" <gi...@apache.org>]

hubcio commented on PR #2925:
URL: https://github.com/apache/iggy/pull/2925#issuecomment-4109845166

   @mlevkov you need to regenerate lockfile


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["spetz (via GitHub)" <gi...@apache.org>]

spetz commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r2986294790


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2121 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: f64 = 2.0;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    Ndjson,
+    /// All messages as a single JSON array.
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    Raw,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2.0).
+    pub retry_backoff_multiplier: Option<f64>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: f64,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<reqwest::Client>,
+    requests_sent: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    retries_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+/// Parse a human-readable duration string, falling back to a default on failure.
+fn parse_duration(input: Option<&str>, default: &str) -> Duration {
+    let raw = input.unwrap_or(default);
+    HumanDuration::from_str(raw)
+        .map(|d| *d)
+        .unwrap_or_else(|e| {
+            warn!(
+                "Invalid duration '{}': {}, using default '{}'",
+                raw, e, default
+            );
+            *HumanDuration::from_str(default).expect("default duration must be valid")
+        })
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1.0);
+        let max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 All retry delays will be capped to max_retry_delay.",
+                id, retry_delay, max_retry_delay,
+            );
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            requests_sent: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            retries_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` from resolved config.
+    fn build_client(&self) -> Result<reqwest::Client, Error> {
+        let builder = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs);
+
+        builder
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))
+    }
+
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self.batch_mode {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::Ndjson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(&bytes),
+                "iggy_payload_encoding": "base64"
+            })),
+            Payload::Proto(proto_str) => Ok(serde_json::json!({
+                "data": general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                "iggy_payload_encoding": "base64"
+            })),
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> serde_json::Value {
+        if !self.include_metadata {
+            return payload_json;
+        }
+
+        let mut metadata = serde_json::json!({
+            "iggy_id": format_u128_as_uuid(message.id),
+            "iggy_offset": message.offset,
+            "iggy_timestamp": message.timestamp,
+            "iggy_stream": topic_metadata.stream,
+            "iggy_topic": topic_metadata.topic,
+            "iggy_partition_id": messages_metadata.partition_id,
+        });
+
+        if self.include_checksum {
+            metadata["iggy_checksum"] = serde_json::json!(message.checksum);

Review Comment:
   Same regarding `iggy_checksum`, `iggy_origin_timestamp` and any other magic strings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(connectors): add generic HTTP sink connector [iggy]

["mlevkov (via GitHub)" <gi...@apache.org>]

mlevkov commented on code in PR #2925:
URL: https://github.com/apache/iggy/pull/2925#discussion_r3019483663


##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,

Review Comment:
   Fixed. Added `sanitize_url_for_log()` that strips userinfo (`user:password@`) before logging. All log/error messages now use `self.log_url` (sanitized); only actual HTTP requests use `self.url` (raw). Falls back to regex-style stripping when `Url::parse` fails (prevents credential leak even for malformed URLs). Good catch on the `user:password@` vector.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]

Review Comment:
   Fixed. Added `#[serde(rename = "ndjson")]` on the `NdJson` variant to override the `snake_case` default. All config files, README, and tests use `"ndjson"`. Added a comment explaining the rename precedence for future maintainers. The Rust variant stays `NdJson` (correct Rust casing), serializes as `"ndjson"` (industry standard per ndjson.org).



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,
+                body.len(),
+            );
+        }
+
+        self.send_attempts.fetch_add(1, Ordering::Relaxed);
+
+        let response = build_request(self.method, client, &self.url)
+            .headers(headers.clone())
+            .header("content-type", content_type)
+            .body(body)
+            .send()
+            .await
+            .map_err(|e| {
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                error!(
+                    "HTTP sink ID: {} — request to {} failed after middleware retries: {:#}",
+                    self.id, self.url, e
+                );
+                Error::HttpRequestFailed(format!("HTTP {} — {}", self.url, e))
+            })?;
+
+        let status = response.status();
+        // success_status_codes is checked in BOTH the retry strategy (to stop retrying)
+        // AND here (to classify the final response). Both must use the same set.
+        if self.success_status_codes.contains(&status.as_u16()) {
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — success (status {})",
+                    self.id,
+                    status.as_u16()
+                );
+            }
+            self.record_success();
+            return Ok(());
+        }
+
+        // Non-success status after middleware exhausted retries — read body for diagnostics
+        let response_body = match response.text().await {
+            Ok(body) => body,
+            Err(e) => format!("<body read error: {}>", e),
+        };
+
+        error!(
+            "HTTP sink ID: {} — request failed (status {}). Response: {}",
+            self.id,
+            status.as_u16(),
+            truncate_response(&response_body, 500),
+        );
+        self.errors_count.fetch_add(1, Ordering::Relaxed);
+        Err(Error::HttpRequestFailed(format!(
+            "HTTP {} — status: {}",
+            self.url,
+            status.as_u16()
+        )))
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    self.batch_mode,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self.send_with_retry(Bytes::from(body), content_type).await {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, self.batch_mode, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id,
+                    self.batch_mode,
+                    delivered,
+                    total,
+                    http_failures,
+                    serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(messages, self.batch_mode.content_type(), |mut message| {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = self.payload_to_json(payload)?;
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json)?;
+            serde_json::to_vec(&envelope)
+                .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+        })
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(&self, body: Bytes, count: u64, skipped: u64) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(body, self.batch_mode.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);
+            }
+            if skipped > 0 {
+                error!(
+                    "HTTP sink ID: {} — {} batch failed with {} serialization skips",
+                    self.id, self.batch_mode, skipped,
+                );
+            }
+            return Err(e);
+        }
+        self.messages_delivered.fetch_add(count, Ordering::Relaxed);
+        if skipped > 0 {
+            warn!(
+                "HTTP sink ID: {} — {} batch: {} delivered, {} skipped (serialization errors)",
+                self.id, self.batch_mode, count, skipped,
+            );
+        }
+        Ok(())
+    }
+
+    /// Send messages in `ndjson` mode — all messages in one request, newline-delimited.
+    /// Skips individual messages that fail serialization rather than aborting the batch.
+    async fn send_ndjson(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        let mut lines = Vec::with_capacity(messages.len());
+        let mut skipped = 0u64;
+
+        for mut message in messages {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = match self.payload_to_json(payload) {
+                Ok(json) => json,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch: {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            let envelope = match self.build_envelope(
+                &message,
+                topic_metadata,
+                messages_metadata,
+                payload_json,
+            ) {
+                Ok(env) => env,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (envelope): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            };
+            match serde_json::to_string(&envelope) {
+                Ok(line) => lines.push(line),
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — skipping message at offset {} in NDJSON batch (serialize): {}",
+                        self.id, message.offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    skipped += 1;
+                    continue;
+                }
+            }
+        }
+
+        if lines.is_empty() {
+            return Err(Error::Serialization(
+                "All messages in NDJSON batch failed serialization".to_string(),
+            ));
+        }
+
+        let count = lines.len() as u64;
+
+        let mut body_str = lines.join("\n");

Review Comment:
   Fixed. Replaced `Vec<String>` + `join("\n")` with direct `serde_json::to_writer(&mut body, &envelope)` into a single pre-sized `Vec<u8>`. Each message is serialized directly into the output buffer followed by `body.push(b'\n')`. On error, `body.truncate(pos)` rolls back the partial write. Eliminates both the intermediate `String` per message and the `join` copy. `Vec::with_capacity(messages.len() * 256)` avoids reallocation for typical payloads.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)
+            .map_err(|e| Error::Serialization(format!("MetadataEnvelope: {}", e)))
+    }
+
+    /// Record a successful request timestamp.
+    fn record_success(&self) {
+        let now = SystemTime::now()
+            .duration_since(UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        self.last_success_timestamp.store(now, Ordering::Relaxed);
+    }
+
+    /// Send an HTTP request with retry via reqwest-middleware. Returns Ok on success,
+    /// Err after exhausting retries. Retry logic (backoff, transient classification)
+    /// is handled by the middleware configured in `build_client()`.
+    async fn send_with_retry(&self, body: Bytes, content_type: &str) -> Result<(), Error> {
+        let client = self.client()?;
+        let headers = self.request_headers.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP headers not initialized — was open() called?".to_string())
+        })?;
+
+        if self.verbose {
+            debug!(
+                "HTTP sink ID: {} — sending {:?} {} ({} bytes)",
+                self.id,
+                self.method,
+                self.url,
+                body.len(),
+            );
+        }
+
+        self.send_attempts.fetch_add(1, Ordering::Relaxed);
+
+        let response = build_request(self.method, client, &self.url)
+            .headers(headers.clone())
+            .header("content-type", content_type)
+            .body(body)
+            .send()
+            .await
+            .map_err(|e| {
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                error!(
+                    "HTTP sink ID: {} — request to {} failed after middleware retries: {:#}",
+                    self.id, self.url, e
+                );
+                Error::HttpRequestFailed(format!("HTTP {} — {}", self.url, e))
+            })?;
+
+        let status = response.status();
+        // success_status_codes is checked in BOTH the retry strategy (to stop retrying)
+        // AND here (to classify the final response). Both must use the same set.
+        if self.success_status_codes.contains(&status.as_u16()) {
+            if self.verbose {
+                debug!(
+                    "HTTP sink ID: {} — success (status {})",
+                    self.id,
+                    status.as_u16()
+                );
+            }
+            self.record_success();
+            return Ok(());
+        }
+
+        // Non-success status after middleware exhausted retries — read body for diagnostics
+        let response_body = match response.text().await {
+            Ok(body) => body,
+            Err(e) => format!("<body read error: {}>", e),
+        };
+
+        error!(
+            "HTTP sink ID: {} — request failed (status {}). Response: {}",
+            self.id,
+            status.as_u16(),
+            truncate_response(&response_body, 500),
+        );
+        self.errors_count.fetch_add(1, Ordering::Relaxed);
+        Err(Error::HttpRequestFailed(format!(
+            "HTTP {} — status: {}",
+            self.url,
+            status.as_u16()
+        )))
+    }
+
+    /// Shared per-message send loop for `individual` and `raw` modes.
+    ///
+    /// Iterates `messages`, builds a body for each via `build_body`, enforces payload size
+    /// limits, sends via `send_with_retry`, and tracks partial delivery.
+    /// Aborts after `MAX_CONSECUTIVE_FAILURES` consecutive HTTP failures.
+    ///
+    /// `build_body` takes ownership of each `ConsumedMessage` — callers must extract
+    /// all needed fields (payload, metadata) within the closure.
+    async fn send_per_message<F>(
+        &self,
+        messages: Vec<ConsumedMessage>,
+        content_type: &str,
+        mut build_body: F,
+    ) -> Result<(), Error>
+    where
+        F: FnMut(ConsumedMessage) -> Result<Vec<u8>, Error>,
+    {
+        let total = messages.len();
+        let mut delivered = 0u64;
+        let mut http_failures = 0u64;
+        let mut serialization_failures = 0u64;
+        let mut consecutive_failures = 0u32;
+        let mut last_error: Option<Error> = None;
+
+        for message in messages {
+            let offset = message.offset;
+            let body = match build_body(message) {
+                Ok(b) => b,
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to build {} body at offset {}: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    self.errors_count.fetch_add(1, Ordering::Relaxed);
+                    serialization_failures += 1;
+                    last_error = Some(e);
+                    continue;
+                }
+            };
+
+            if self.max_payload_size_bytes > 0 && body.len() as u64 > self.max_payload_size_bytes {
+                error!(
+                    "HTTP sink ID: {} — {} payload at offset {} exceeds max size ({} > {} bytes). Skipping.",
+                    self.id,
+                    self.batch_mode,
+                    offset,
+                    body.len(),
+                    self.max_payload_size_bytes,
+                );
+                self.errors_count.fetch_add(1, Ordering::Relaxed);
+                serialization_failures += 1;
+                last_error = Some(Error::HttpRequestFailed(format!(
+                    "Payload exceeds max size: {} bytes",
+                    body.len()
+                )));
+                continue;
+            }
+
+            match self.send_with_retry(Bytes::from(body), content_type).await {
+                Ok(()) => {
+                    delivered += 1;
+                    consecutive_failures = 0;
+                }
+                Err(e) => {
+                    error!(
+                        "HTTP sink ID: {} — failed to deliver {} message at offset {} after retries: {}",
+                        self.id, self.batch_mode, offset, e
+                    );
+                    http_failures += 1;
+                    consecutive_failures += 1;
+                    last_error = Some(e);
+
+                    if consecutive_failures >= MAX_CONSECUTIVE_FAILURES {
+                        let processed = delivered + http_failures + serialization_failures;
+                        debug_assert!(
+                            processed <= total as u64,
+                            "processed ({processed}) > total ({total}) — accounting bug"
+                        );
+                        let skipped = (total as u64).saturating_sub(processed);
+                        error!(
+                            "HTTP sink ID: {} — aborting {} batch after {} consecutive HTTP failures \
+                             ({} remaining messages skipped)",
+                            self.id, self.batch_mode, consecutive_failures, skipped,
+                        );
+                        self.errors_count.fetch_add(skipped, Ordering::Relaxed);
+                        break;
+                    }
+                }
+            }
+        }
+
+        self.messages_delivered
+            .fetch_add(delivered, Ordering::Relaxed);
+
+        match last_error {
+            Some(e) => {
+                error!(
+                    "HTTP sink ID: {} — partial {} delivery: {}/{} delivered, \
+                     {} HTTP failures, {} serialization errors",
+                    self.id,
+                    self.batch_mode,
+                    delivered,
+                    total,
+                    http_failures,
+                    serialization_failures,
+                );
+                Err(e)
+            }
+            None => Ok(()),
+        }
+    }
+
+    /// Send messages in `individual` mode — one HTTP request per message.
+    async fn send_individual(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        self.send_per_message(messages, self.batch_mode.content_type(), |mut message| {
+            let payload = std::mem::replace(&mut message.payload, Payload::Raw(vec![]));
+            let payload_json = self.payload_to_json(payload)?;
+            let envelope =
+                self.build_envelope(&message, topic_metadata, messages_metadata, payload_json)?;
+            serde_json::to_vec(&envelope)
+                .map_err(|e| Error::Serialization(format!("Envelope serialize: {}", e)))
+        })
+        .await
+    }
+
+    /// Sends a batch body and updates delivery/error accounting.
+    ///
+    /// Shared by `send_ndjson` and `send_json_array` — the post-send accounting logic
+    /// (error propagation, skip warnings) is identical across batch modes.
+    async fn send_batch_body(&self, body: Bytes, count: u64, skipped: u64) -> Result<(), Error> {
+        debug_assert!(
+            count > 0,
+            "send_batch_body called with count=0 — callers must guard against empty batches"
+        );
+        if let Err(e) = self
+            .send_with_retry(body, self.batch_mode.content_type())
+            .await
+        {
+            // send_with_retry already added 1 to errors_count for the HTTP failure.
+            // Add the remaining messages that were serialized but not delivered.
+            if count > 1 {
+                self.errors_count.fetch_add(count - 1, Ordering::Relaxed);

Review Comment:
   Fixed. Changed `count - 1` to `count.saturating_sub(1)` and added an INVARIANT comment documenting the dependency on `send_with_retry` incrementing `errors_count` exactly once before returning `Err` (with line references to both error paths). The `debug_assert!(count > 0)` at the top of the function catches the zero case in dev builds.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {
+    iggy_id: String,
+    iggy_offset: u64,
+    iggy_timestamp: u64,
+    iggy_stream: String,
+    iggy_topic: String,
+    iggy_partition_id: u32,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_checksum: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_origin_timestamp: Option<u64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    iggy_headers: Option<serde_json::Map<String, serde_json::Value>>,
+}
+
+/// Binary payload with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedPayload {
+    data: String,
+    iggy_payload_encoding: &'static str,
+}
+
+/// Binary header value with base64 encoding marker.
+#[derive(Debug, Serialize)]
+struct EncodedHeader {
+    data: String,
+    iggy_header_encoding: &'static str,
+}
+
+/// Configuration for the HTTP sink connector, deserialized from [plugin_config] in config.toml.
+#[derive(Debug, Serialize, Deserialize)]
+pub struct HttpSinkConfig {
+    /// Target URL for HTTP requests (required).
+    pub url: String,
+    /// HTTP method (default: POST).
+    pub method: Option<HttpMethod>,
+    /// Request timeout as a human-readable duration string, e.g. "30s" (default: 30s).
+    pub timeout: Option<String>,
+    /// Maximum HTTP body size in bytes (default: 10MB). Set to 0 to disable.
+    pub max_payload_size_bytes: Option<u64>,
+    /// Custom HTTP headers.
+    pub headers: Option<HashMap<String, String>>,
+    /// Payload formatting mode (default: individual).
+    pub batch_mode: Option<BatchMode>,
+    /// Include Iggy metadata envelope in payload (default: true).
+    pub include_metadata: Option<bool>,
+    /// Include message checksum in metadata (default: false).
+    pub include_checksum: Option<bool>,
+    /// Include origin timestamp in metadata (default: false).
+    pub include_origin_timestamp: Option<bool>,
+    /// Enable health check request in open() (default: false).
+    pub health_check_enabled: Option<bool>,
+    /// HTTP method for health check (default: HEAD).
+    pub health_check_method: Option<HttpMethod>,
+    /// Maximum number of retries for transient errors (default: 3).
+    pub max_retries: Option<u32>,
+    /// Retry delay as a human-readable duration string, e.g. "1s" (default: 1s).
+    pub retry_delay: Option<String>,
+    /// Backoff multiplier for exponential retry delay (default: 2).
+    pub retry_backoff_multiplier: Option<u32>,
+    /// Maximum retry delay cap as a human-readable duration string (default: 30s).
+    pub max_retry_delay: Option<String>,
+    /// HTTP status codes considered successful (default: [200, 201, 202, 204]).
+    pub success_status_codes: Option<Vec<u16>>,
+    /// Accept invalid TLS certificates (default: false). Named to signal danger.
+    pub tls_danger_accept_invalid_certs: Option<bool>,
+    /// Maximum idle connections per host (default: 10).
+    pub max_connections: Option<usize>,
+    /// Enable verbose request/response logging (default: false).
+    pub verbose_logging: Option<bool>,
+}
+
+/// HTTP sink connector that delivers consumed messages to any HTTP endpoint.
+///
+/// Lifecycle: `new()` → `open()` → `consume()` (repeated) → `close()`.
+/// The `reqwest::Client` is built in `open()` (not `new()`) so that config-derived
+/// settings (timeout, TLS, connection pool) are applied. This matches the
+/// MongoDB/Elasticsearch/PostgreSQL sink initialization pattern.
+#[derive(Debug)]
+pub struct HttpSink {
+    id: u32,
+    url: String,
+    method: HttpMethod,
+    timeout: Duration,
+    max_payload_size_bytes: u64,
+    headers: HashMap<String, String>,
+    batch_mode: BatchMode,
+    include_metadata: bool,
+    include_checksum: bool,
+    include_origin_timestamp: bool,
+    health_check_enabled: bool,
+    health_check_method: HttpMethod,
+    max_retries: u32,
+    retry_delay: Duration,
+    retry_backoff_multiplier: u32,
+    max_retry_delay: Duration,
+    success_status_codes: HashSet<u16>,
+    tls_danger_accept_invalid_certs: bool,
+    max_connections: usize,
+    verbose: bool,
+    /// Pre-built HTTP headers (excluding Content-Type). Built once in `open()` from validated
+    /// `self.headers`, reused for every request. `None` before `open()` is called.
+    request_headers: Option<reqwest::header::HeaderMap>,
+    /// Initialized in `open()` with config-derived settings. `None` before `open()` is called.
+    client: Option<ClientWithMiddleware>,
+    send_attempts: AtomicU64,
+    messages_delivered: AtomicU64,
+    errors_count: AtomicU64,
+    /// Epoch seconds of last successful HTTP request.
+    last_success_timestamp: AtomicU64,
+}
+
+impl HttpSink {
+    pub fn new(id: u32, config: HttpSinkConfig) -> Self {
+        let url = config.url;
+        let method = config.method.unwrap_or_default();
+        let timeout = parse_duration(config.timeout.as_deref(), DEFAULT_TIMEOUT);
+        let max_payload_size_bytes = config
+            .max_payload_size_bytes
+            .unwrap_or(DEFAULT_MAX_PAYLOAD_SIZE);
+        let headers = config.headers.unwrap_or_default();
+        let batch_mode = config.batch_mode.unwrap_or_default();
+        let include_metadata = config.include_metadata.unwrap_or(true);
+        let include_checksum = config.include_checksum.unwrap_or(false);
+        let include_origin_timestamp = config.include_origin_timestamp.unwrap_or(false);
+        let health_check_enabled = config.health_check_enabled.unwrap_or(false);
+        let health_check_method = config.health_check_method.unwrap_or(HttpMethod::Head);
+        let max_retries = config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES);
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let retry_backoff_multiplier = config
+            .retry_backoff_multiplier
+            .unwrap_or(DEFAULT_BACKOFF_MULTIPLIER)
+            .max(1);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        let success_status_codes: HashSet<u16> = config
+            .success_status_codes
+            .unwrap_or_else(|| vec![200, 201, 202, 204])
+            .into_iter()
+            .collect();
+        let tls_danger_accept_invalid_certs =
+            config.tls_danger_accept_invalid_certs.unwrap_or(false);
+        let max_connections = config.max_connections.unwrap_or(DEFAULT_MAX_CONNECTIONS);
+        let verbose = config.verbose_logging.unwrap_or(false);
+
+        if retry_delay > max_retry_delay {
+            warn!(
+                "HTTP sink ID: {} — retry_delay ({:?}) exceeds max_retry_delay ({:?}). \
+                 Swapping values to prevent ExponentialBackoff panic.",
+                id, retry_delay, max_retry_delay,
+            );
+            std::mem::swap(&mut retry_delay, &mut max_retry_delay);
+        }
+
+        if tls_danger_accept_invalid_certs {
+            warn!(
+                "HTTP sink ID: {} — tls_danger_accept_invalid_certs is enabled. \
+                 TLS certificate validation is DISABLED.",
+                id
+            );
+        }
+
+        if batch_mode == BatchMode::Raw && include_metadata {
+            warn!(
+                "HTTP sink ID: {} — batch_mode=raw ignores include_metadata. \
+                 Raw mode sends payload bytes directly without metadata envelope.",
+                id
+            );
+        }
+
+        if matches!(method, HttpMethod::Get | HttpMethod::Head)
+            && batch_mode != BatchMode::Individual
+        {
+            warn!(
+                "HTTP sink ID: {} — {:?} with batch_mode={:?} will send a request body. \
+                 Some servers may reject GET/HEAD requests with a body.",
+                id, method, batch_mode,
+            );
+        }
+
+        HttpSink {
+            id,
+            url,
+            method,
+            timeout,
+            max_payload_size_bytes,
+            headers,
+            batch_mode,
+            include_metadata,
+            include_checksum,
+            include_origin_timestamp,
+            health_check_enabled,
+            health_check_method,
+            max_retries,
+            retry_delay,
+            retry_backoff_multiplier,
+            max_retry_delay,
+            success_status_codes,
+            tls_danger_accept_invalid_certs,
+            max_connections,
+            verbose,
+            request_headers: None,
+            client: None,
+            send_attempts: AtomicU64::new(0),
+            messages_delivered: AtomicU64::new(0),
+            errors_count: AtomicU64::new(0),
+            last_success_timestamp: AtomicU64::new(0),
+        }
+    }
+
+    /// Build the `reqwest::Client` wrapped with retry and tracing middleware.
+    fn build_client(&self) -> Result<ClientWithMiddleware, Error> {
+        let raw_client = reqwest::Client::builder()
+            .timeout(self.timeout)
+            .pool_max_idle_per_host(self.max_connections)
+            .pool_idle_timeout(Duration::from_secs(DEFAULT_POOL_IDLE_TIMEOUT_SECS))
+            .tcp_keepalive(Duration::from_secs(DEFAULT_TCP_KEEPALIVE_SECS))
+            .danger_accept_invalid_certs(self.tls_danger_accept_invalid_certs)
+            .build()
+            .map_err(|e| Error::InitError(format!("Failed to build HTTP client: {}", e)))?;
+
+        let retry_policy = ExponentialBackoff::builder()
+            .retry_bounds(self.retry_delay, self.max_retry_delay)
+            .base(self.retry_backoff_multiplier)
+            .build_with_max_retries(self.max_retries);
+
+        let retry_strategy = HttpSinkRetryStrategy {
+            success_status_codes: self.success_status_codes.clone(),
+        };
+
+        let retry_middleware =
+            RetryTransientMiddleware::new_with_policy_and_strategy(retry_policy, retry_strategy);
+
+        Ok(ClientBuilder::new(raw_client)
+            .with(TracingMiddleware::default())
+            .with(retry_middleware)
+            .build())
+    }
+
+    /// Returns the initialized HTTP client, or an error if `open()` was not called.
+    fn client(&self) -> Result<&ClientWithMiddleware, Error> {
+        self.client.as_ref().ok_or_else(|| {
+            Error::InitError("HTTP client not initialized — was open() called?".to_string())
+        })
+    }
+
+    /// Convert a `Payload` to a JSON value for metadata wrapping.
+    /// Non-JSON payloads are base64-encoded with a `iggy_payload_encoding` marker.
+    ///
+    /// Note: All current `Payload` variants produce infallible conversions.
+    /// The `Result` return type exists as a safety net for future variants.
+    fn payload_to_json(&self, payload: Payload) -> Result<serde_json::Value, Error> {
+        match payload {
+            Payload::Json(value) => {
+                // Direct structural conversion (not serialization roundtrip).
+                // Follows the Elasticsearch sink pattern. NaN/Infinity f64 → null.
+                Ok(owned_value_to_serde_json(&value))
+            }
+            Payload::Text(text) => Ok(serde_json::Value::String(text)),
+            Payload::Raw(bytes) | Payload::FlatBuffer(bytes) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(&bytes),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+            Payload::Proto(proto_str) => {
+                let encoded = EncodedPayload {
+                    data: general_purpose::STANDARD.encode(proto_str.as_bytes()),
+                    iggy_payload_encoding: ENCODING_BASE64,
+                };
+                serde_json::to_value(encoded)
+                    .map_err(|e| Error::Serialization(format!("EncodedPayload: {}", e)))
+            }
+        }
+    }
+
+    /// Build a message envelope with optional metadata wrapping.
+    fn build_envelope(
+        &self,
+        message: &ConsumedMessage,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: &MessagesMetadata,
+        payload_json: serde_json::Value,
+    ) -> Result<serde_json::Value, Error> {
+        if !self.include_metadata {
+            return Ok(payload_json);
+        }
+
+        let headers_map = if let Some(ref headers) = message.headers
+            && !headers.is_empty()
+        {
+            let map: serde_json::Map<String, serde_json::Value> = headers
+                .iter()
+                .map(|(k, v)| {
+                    // Raw bytes: base64-encode to avoid Rust debug format in JSON output.
+                    // as_raw() returns Ok only for HeaderKind::Raw.
+                    let value = if let Ok(raw) = v.as_raw() {
+                        let encoded = EncodedHeader {
+                            data: general_purpose::STANDARD.encode(raw),
+                            iggy_header_encoding: ENCODING_BASE64,
+                        };
+                        serde_json::to_value(encoded)
+                            .map_err(|e| Error::Serialization(format!("EncodedHeader: {}", e)))?
+                    } else {
+                        serde_json::Value::String(v.to_string_value())
+                    };
+                    Ok((k.to_string_value(), value))
+                })
+                .collect::<Result<serde_json::Map<String, serde_json::Value>, Error>>()?;
+            Some(map)
+        } else {
+            None
+        };
+
+        let metadata = IggyMetadata {
+            iggy_id: format_u128_as_hex(message.id),
+            iggy_offset: message.offset,
+            iggy_timestamp: message.timestamp,
+            iggy_stream: topic_metadata.stream.clone(),
+            iggy_topic: topic_metadata.topic.clone(),
+            iggy_partition_id: messages_metadata.partition_id,
+            iggy_checksum: if self.include_checksum {
+                Some(message.checksum)
+            } else {
+                None
+            },
+            iggy_origin_timestamp: if self.include_origin_timestamp {
+                Some(message.origin_timestamp)
+            } else {
+                None
+            },
+            iggy_headers: headers_map,
+        };
+
+        let envelope = MetadataEnvelope {
+            metadata,
+            payload: payload_json,
+        };
+
+        serde_json::to_value(envelope)

Review Comment:
   Addressed. Initially added `build_envelope_vec` and `build_envelope_to_writer` to eliminate the `Value` intermediate, but 6-agent review flagged the triple maintenance surface as a higher risk than the ~microsecond serialization overhead. Collapsed back to single `build_envelope` entry point — callers choose their serialization format (`to_vec` for individual, `to_writer` for ndjson, `Value` directly for json_array). `build_iggy_metadata` remains as the shared helper. The performance difference is negligible against HTTP round-trip cost.



##########
core/connectors/sinks/http_sink/src/lib.rs:
##########
@@ -0,0 +1,2069 @@
+/* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use async_trait::async_trait;
+use base64::Engine;
+use base64::engine::general_purpose;
+use bytes::Bytes;
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{
+    ConsumedMessage, Error, MessagesMetadata, Payload, Sink, TopicMetadata,
+    convert::owned_value_to_serde_json, sink_connector,
+};
+use reqwest_middleware::{ClientBuilder, ClientWithMiddleware};
+use reqwest_retry::{
+    RetryTransientMiddleware, Retryable, RetryableStrategy, policies::ExponentialBackoff,
+};
+use reqwest_tracing::TracingMiddleware;
+use serde::{Deserialize, Serialize};
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
+use tracing::{debug, error, info, warn};
+
+sink_connector!(HttpSink);
+
+const DEFAULT_TIMEOUT: &str = "30s";
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_BACKOFF_MULTIPLIER: u32 = 2;
+const DEFAULT_MAX_PAYLOAD_SIZE: u64 = 10 * 1024 * 1024; // 10 MB
+const DEFAULT_MAX_CONNECTIONS: usize = 10;
+/// TCP keep-alive interval for detecting dead connections behind load balancers.
+/// Cloud LBs silently drop idle connections (AWS ALB ~60s, GCP ~600s);
+/// probing at 30s detects these before requests fail.
+const DEFAULT_TCP_KEEPALIVE_SECS: u64 = 30;
+/// Close pooled connections unused for this long. Prevents stale connections
+/// from accumulating when traffic is bursty.
+const DEFAULT_POOL_IDLE_TIMEOUT_SECS: u64 = 90;
+/// Abort remaining messages in individual/raw mode after this many consecutive HTTP failures.
+/// Prevents hammering a dead endpoint with N sequential retry cycles per poll.
+const MAX_CONSECUTIVE_FAILURES: u32 = 3;
+
+const ENCODING_BASE64: &str = "base64";
+
+/// HTTP method enum — validated at deserialization, prevents invalid values like "DELEET" or "GETX".
+#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "UPPERCASE")]
+pub enum HttpMethod {
+    Get,
+    Head,
+    #[default]
+    Post,
+    Put,
+    Patch,
+    Delete,
+}
+
+/// Payload formatting mode for HTTP requests.
+#[derive(
+    Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, strum_macros::Display,
+)]
+#[serde(rename_all = "snake_case")]
+pub enum BatchMode {
+    /// One HTTP request per message (default). Note: with batch_length=50, this produces 50
+    /// sequential HTTP round trips per poll cycle. Use ndjson or json_array for higher throughput.
+    #[default]
+    #[strum(to_string = "individual")]
+    Individual,
+    /// All messages in one request, newline-delimited JSON.
+    #[strum(to_string = "NDJSON")]
+    NdJson,
+    /// All messages as a single JSON array.
+    #[strum(to_string = "JSON array")]
+    JsonArray,
+    /// Raw bytes, one request per message (for non-JSON payloads).
+    #[strum(to_string = "raw")]
+    Raw,
+}
+
+impl BatchMode {
+    /// Determine the Content-Type header based on batch mode.
+    fn content_type(&self) -> &'static str {
+        match self {
+            BatchMode::Individual | BatchMode::JsonArray => "application/json",
+            BatchMode::NdJson => "application/x-ndjson",
+            BatchMode::Raw => "application/octet-stream",
+        }
+    }
+}
+
+/// Metadata envelope wrapping a payload with Iggy message metadata.
+#[derive(Debug, Serialize)]
+struct MetadataEnvelope {
+    metadata: IggyMetadata,
+    payload: serde_json::Value,
+}
+
+/// Iggy message metadata fields.
+#[derive(Debug, Serialize)]
+struct IggyMetadata {

Review Comment:
   Added a doc comment on `IggyMetadata` documenting the intentional divergence: HTTP sink uses `iggy_*` (standard JSON convention), ES sink uses `_iggy_*` (ES internal field convention). Both are defensible for their respective ecosystems. Agree a follow-up to unify would be valuable but it touches another component — happy to do it as a separate PR if the team has a preferred convention.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org