You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "kparisa (via GitHub)" <gi...@apache.org> on 2026/10/03 09:37:24 UTC

[PR] feat(connectors): add Google BigQuery sink connector [iggy]

kparisa opened a new pull request, #4394:
URL: https://github.com/apache/iggy/pull/4394

   ## Which issue does this PR address?
   Closes #4393
   
   ## Rationale
   Streams Iggy messages into BigQuery through the Storage Write API _default stream with Arrow-encoded rows. Supports mapped and raw modes, Iggy metadata columns, per-row rejection with offset logging, request splitting below the AppendRows limit, and retries for transient gRPC failures. Verified against a live BigQuery table; CI coverage uses an in-process fake of tables.get and BigQueryWrite.
   
   
   ## What changed?
   New sink connector under /core/connectors/sinks
   
   ## Local Execution
   - Passed 
   - Pre-commit hooks ran
   
   ## AI Usage
   - Leveraged ChatGPT, Claude for planning, test scenarios, implementation.
   - Ran local tests, publishing payloads to iggy, configured GCP test project, created the BigQuery (BQ) dataset "iggy" and events table. 
   - Verified the records in the BQ table.
   - Tested failure scenarios as well for mis-configs and missing table definitions etc
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4394:
URL: https://github.com/apache/iggy/pull/4394#issuecomment-5981233723

   /author


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio merged PR #4394:
URL: https://github.com/apache/iggy/pull/4394


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4173903106


##########
core/connectors/sinks/bigquery_sink/src/lib.rs:
##########
@@ -0,0 +1,457 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Google BigQuery sink connector.
+//!
+//! Writes Iggy messages into a BigQuery table through the Storage Write API
+//! `_default` stream, with rows encoded as Arrow record batches.
+//!
+//! Module map:
+//! - [`schema`]: BigQuery table schema (from `tables.get`) to Arrow schema.

Review Comment:
   warning: The crate docs link five private modules, so rustdoc warns and the doctest job fails on `-D warnings`. Remove the square brackets around `schema`, `encode`, `client`, `error` and `sink`.



##########
core/connectors/sinks/bigquery_sink/src/error.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Failure classification for BigQuery calls.
+//!
+//! Only transient failures are retried. Retrying a permanent one (bad
+//! schema, missing permission) cannot succeed and only delays the error.
+
+use crate::schema::SchemaError;
+use gcloud_gax::grpc::{Code, Status};
+use iggy_connector_sdk::Error;
+use std::fmt;
+
+/// An `AppendRows` call that did not append. Row-level errors are not part
+/// of this type: they are an expected outcome the caller handles by
+/// dropping the reported rows.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum AppendError {
+    /// The call failed, or the response carried an error status.
+    Rpc { code: Code, message: String },
+    /// The response stream closed without a response.
+    NoResponse,
+}
+
+/// A `tables.get` call that did not return a usable schema.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum TableError {
+    Http { status: u16, body: String },
+    Transport(String),
+    Token(String),
+    Schema(SchemaError),
+}
+
+impl AppendError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            AppendError::Rpc { code, .. } => is_retryable_code(*code),
+            // The append may or may not have landed. Retrying risks a
+            // duplicate, which the delivery guarantees allow.
+            AppendError::NoResponse => true,
+        }
+    }
+}
+
+impl TableError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            TableError::Http { status, .. } => *status == 429 || *status >= 500,
+            TableError::Transport(_) => true,
+            TableError::Token(_) | TableError::Schema(_) => false,

Review Comment:
   warning: `TableError::Token` is not retryable, so `retry_async` aborts `open()` on the first attempt when a token refresh fails. Return `true` for `TableError::Token` so the connector retries transient token failures.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, the run is decoded again row by
+//! row so that only the offending rows are dropped.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, TimestampMicrosecondArray,
+};
+use arrow::compute::{cast, concat_batches};
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, kept.len(), |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(to_hex(&bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    row_count: usize,
+    mut on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() {
+        return Ok((None, (0..row_count).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    if let Ok(batch) = decode(&schema, rows) {
+        return Ok((Some(to_target(batch, &target)?), (0..row_count).collect()));
+    }
+
+    let mut batches = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in rows.iter().enumerate() {
+        match decode(&schema, std::slice::from_ref(row)) {
+            Ok(batch) => {
+                batches.push(batch);
+                survivors.push(index);
+            }
+            Err(e) => on_reject(index, format!("row does not match the table schema: {e}")),
+        }
+    }
+    if batches.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = concat_batches(&schema, &batches)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    decoder.serialize(rows)?;
+    decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut values = Vec::with_capacity(messages.len());
+            for message in messages {
+                match message.payload.try_to_bytes() {
+                    Ok(bytes) => {
+                        values.push(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(BinaryArray::from_iter_values(values))
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_owned_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => Arc::new(StringArray::from_iter_values(
+                messages.iter().map(|m| m.id.to_string()),
+            )),
+            MetaColumn::Headers => {
+                let required = !field.is_nullable();
+                Arc::new(StringArray::from_iter(
+                    messages.iter().map(|m| headers_json(m, required)),
+                ))
+            }
+        };
+        fields.push(field.clone());
+        columns.push(column);
+    }
+    Ok(())
+}
+
+/// Headers as a JSON object. Text values are written as strings, binary
+/// values as `{"data": <base64>, "iggy_header_encoding": "base64"}`, the same
+/// shape `http_sink` uses.
+fn headers_json(message: &ConsumedMessage, required: bool) -> Option<String> {
+    let headers = message.headers.as_ref().filter(|h| !h.is_empty());
+    let Some(headers) = headers else {
+        return required.then(|| "{}".to_owned());
+    };
+    let mut object = serde_json::Map::with_capacity(headers.len());
+    for (key, value) in headers {
+        let encoded = match value.as_raw() {
+            Ok(raw) => serde_json::json!({
+                "data": BASE64.encode(raw),
+                "iggy_header_encoding": HEADER_ENCODING_BASE64,
+            }),
+            Err(_) => serde_json::Value::String(value.to_string_value()),
+        };
+        object.insert(key.to_string_value(), encoded);
+    }
+    Some(serde_json::Value::Object(object).to_string())
+}
+
+// ─── Request sizing ──────────────────────────────────────────────────────────
+
+/// Encode `batch` and halve it until every piece fits `max_bytes`. A single
+/// row that does not fit is rejected.
+fn split_into_chunks(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<(), ArrowError> {
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;

Review Comment:
   warning: `split_into_chunks` encodes the batch, discards those bytes when the batch exceeds the budget, and re-encodes both halves, so every split level encodes its part again. Split on an estimated row size and encode each chunk once.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, the run is decoded again row by
+//! row so that only the offending rows are dropped.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, TimestampMicrosecondArray,
+};
+use arrow::compute::{cast, concat_batches};
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, kept.len(), |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(to_hex(&bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    row_count: usize,
+    mut on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() {
+        return Ok((None, (0..row_count).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    if let Ok(batch) = decode(&schema, rows) {
+        return Ok((Some(to_target(batch, &target)?), (0..row_count).collect()));
+    }
+
+    let mut batches = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in rows.iter().enumerate() {
+        match decode(&schema, std::slice::from_ref(row)) {
+            Ok(batch) => {
+                batches.push(batch);
+                survivors.push(index);
+            }
+            Err(e) => on_reject(index, format!("row does not match the table schema: {e}")),
+        }
+    }
+    if batches.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = concat_batches(&schema, &batches)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    decoder.serialize(rows)?;
+    decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut values = Vec::with_capacity(messages.len());
+            for message in messages {
+                match message.payload.try_to_bytes() {
+                    Ok(bytes) => {
+                        values.push(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(BinaryArray::from_iter_values(values))
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_owned_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => Arc::new(StringArray::from_iter_values(
+                messages.iter().map(|m| m.id.to_string()),

Review Comment:
   nit: `append_metadata` allocates one `String` per message id for the `iggy_id` column, and `include_metadata` defaults to true. Build that column with a `StringBuilder` instead.



##########
core/connectors/sinks/bigquery_sink/src/lib.rs:
##########
@@ -0,0 +1,457 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Google BigQuery sink connector.
+//!
+//! Writes Iggy messages into a BigQuery table through the Storage Write API
+//! `_default` stream, with rows encoded as Arrow record batches.
+//!
+//! Module map:
+//! - [`schema`]: BigQuery table schema (from `tables.get`) to Arrow schema.
+//! - [`encode`]: `ConsumedMessage`s to Arrow record batches, split by size.
+//! - [`client`]: credentials, `tables.get`, and `AppendRows` calls.
+//! - [`error`]: gRPC status classification into retryable and permanent.
+//! - [`sink`]: the `Sink` trait implementation.
+
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{Error, sink_connector};
+use secrecy::SecretString;
+use serde::Deserialize;
+use std::str::FromStr;
+use std::sync::atomic::AtomicU64;
+use std::time::Duration;
+use tracing::warn;
+
+mod client;
+mod encode;
+mod error;
+mod schema;
+mod sink;
+
+sink_connector!(BigQuerySink);
+
+const DEFAULT_PAYLOAD_COLUMN: &str = "payload";
+const DEFAULT_MAX_REQUEST_BYTES: usize = 8 * 1024 * 1024;
+/// Hard ceiling below the 10 MB `AppendRows` limit, leaving room for the
+/// request envelope (stream name, writer schema, protobuf framing).
+const MAX_REQUEST_BYTES_CEILING: usize = 9 * 1024 * 1024;
+/// Smallest accepted request budget. Anything lower makes most real rows
+/// unwritable.
+const MIN_REQUEST_BYTES: usize = 64 * 1024;
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_TIMEOUT: &str = "30s";
+
+/// Plugin configuration, deserialized from `[plugin_config]`.
+///
+/// `Deserialize` only. Nothing re-serializes a plugin config, and leaving
+/// `Serialize` off is what keeps `credentials_json` unserializable.
+#[derive(Debug, Clone, Deserialize)]
+pub struct BigQuerySinkConfig {
+    pub project_id: String,
+    pub dataset: String,
+    pub table: String,
+    /// `mapped` (default) or `raw`.
+    pub mode: Option<WriteMode>,
+    /// Path to a service account key file. Mutually exclusive with
+    /// `credentials_json`. When neither is set, Application Default
+    /// Credentials are used.
+    pub credentials_path: Option<String>,
+    /// Inline service account key JSON.
+    pub credentials_json: Option<SecretString>,
+    /// Column holding the payload in `raw` mode (default `payload`).
+    pub payload_column: Option<String>,
+    /// Write `iggy_stream`, `iggy_topic`, `iggy_partition_id`, `iggy_offset`,
+    /// `iggy_timestamp` and `iggy_id` (default `true`).
+    pub include_metadata: Option<bool>,
+    /// Write message headers into an `iggy_headers` column (default `false`).
+    pub include_headers: Option<bool>,
+    /// How BigQuery fills columns absent from a request: `default` (column
+    /// default value, else NULL) or `null`.
+    pub missing_value: Option<MissingValue>,
+    /// Upper bound for one `AppendRows` request (default 8 MiB).
+    pub max_request_bytes: Option<usize>,
+    /// Total attempts per append, including the first (default 3).
+    pub max_retries: Option<u32>,
+    /// Base retry delay, e.g. `"1s"` (default `1s`).
+    pub retry_delay: Option<String>,
+    /// Upper bound for a single retry delay (default `30s`).
+    pub max_retry_delay: Option<String>,
+    /// Per-request timeout for `tables.get` and `AppendRows` (default `30s`).
+    pub timeout: Option<String>,
+    /// Log every batch at info level instead of debug (default `false`).
+    pub verbose_logging: Option<bool>,
+    /// REST endpoint override, e.g. `http://127.0.0.1:9050`. Must be set
+    /// together with `grpc_endpoint`; when both are set no credentials are
+    /// used. Intended for local fakes and emulators.
+    pub endpoint: Option<String>,
+    /// gRPC `host:port` override for the Storage Write API.
+    pub grpc_endpoint: Option<String>,
+}
+
+/// How message payloads map onto table columns.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum WriteMode {
+    /// Top-level JSON fields map to table columns by name.
+    #[default]
+    Mapped,
+    /// The whole payload goes into one column.
+    Raw,
+}
+
+/// Maps to `AppendRowsRequest.default_missing_value_interpretation`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum MissingValue {
+    #[default]
+    Default,
+    Null,
+}
+
+impl MissingValue {
+    /// Wire value of `AppendRowsRequest.MissingValueInterpretation`.
+    pub(crate) fn as_proto(self) -> i32 {
+        match self {
+            MissingValue::Null => 1,
+            MissingValue::Default => 2,
+        }
+    }
+}
+
+/// Resolved, validated runtime settings. Built once in `new()`.
+#[derive(Debug, Clone)]
+pub(crate) struct Settings {
+    pub mode: WriteMode,
+    pub payload_column: String,
+    pub include_metadata: bool,
+    pub include_headers: bool,
+    pub missing_value: MissingValue,
+    pub max_request_bytes: usize,
+    pub max_retries: u32,
+    pub retry_delay: Duration,
+    pub max_retry_delay: Duration,
+    pub timeout: Duration,
+    pub verbose: bool,
+}
+
+/// Counters updated on the hot path without taking a lock.
+#[derive(Debug, Default)]
+pub(crate) struct Counters {
+    pub rows_written: AtomicU64,
+    pub rows_rejected: AtomicU64,
+    pub rows_failed: AtomicU64,
+}
+
+#[derive(Debug)]
+pub struct BigQuerySink {
+    id: u32,
+    config: BigQuerySinkConfig,
+    settings: Settings,
+    /// `project.dataset.table`, safe to log.
+    target: String,
+    client: Option<client::BigQueryClient>,

Review Comment:
   simplification: `BigQuerySink` stores `client` and `layout` in two `Option` fields, but `try_open` sets both and `close` clears both, so neither can exist alone. Store one `Option<​(BigQueryClient, TableLayout)>` instead.



##########
core/connectors/sinks/bigquery_sink/src/sink.rs:
##########
@@ -0,0 +1,299 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::BigQuerySink;
+use crate::client::{AppendOutcome, BigQueryClient};
+use crate::encode::{self, Chunk, RunContext};
+use crate::error::{AppendError, TableError};
+use crate::schema::{TableLayout, parse_table_schema};
+use async_trait::async_trait;
+use iggy_connector_sdk::retry::{RetryPolicy, retry_async};
+use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata};
+use std::sync::atomic::Ordering;
+use tracing::{debug, error, info, warn};
+
+#[async_trait]
+impl Sink for BigQuerySink {
+    async fn open(&mut self) -> Result<(), Error> {
+        info!(
+            "Opening BigQuery sink connector ID: {} for table {} (mode: {:?})",
+            self.id, self.target, self.settings.mode
+        );
+        // The SDK turns a failed open into a bare status code, so this is the
+        // only place the reason reaches the runtime log.
+        self.try_open().await.inspect_err(|e| {
+            error!(
+                "Failed to open BigQuery sink connector ID: {} for table {}: {e}",
+                self.id, self.target
+            );
+        })
+    }
+
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        if messages.is_empty() {
+            return Ok(());
+        }
+        let (Some(client), Some(layout)) = (&self.client, &self.layout) else {
+            return Err(Error::InitError("BigQuery sink is not open".into()));
+        };
+        let received = messages.len();
+        debug!(
+            "BigQuery sink ID: {} received {received} messages from {}/{} partition {} current_offset {}",
+            self.id,
+            topic_metadata.stream,
+            topic_metadata.topic,
+            messages_metadata.partition_id,
+            messages_metadata.current_offset
+        );
+
+        let ctx = RunContext {
+            layout,
+            topic: topic_metadata,
+            messages: &messages_metadata,
+            max_request_bytes: self.settings.max_request_bytes,
+        };
+        let encoded = encode::encode(&ctx, messages).map_err(|e| {
+            self.counters
+                .rows_failed
+                .fetch_add(received as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} cannot encode {received} messages from {}/{} partition {}: {e}",
+                self.id, topic_metadata.stream, topic_metadata.topic, messages_metadata.partition_id
+            );
+            Error::Serialization(e.to_string())
+        })?;
+
+        for rejected in &encoded.rejected {
+            warn!(
+                "BigQuery sink ID: {} dropped message stream={} topic={} partition={} offset={}: {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                rejected.offset,
+                rejected.reason
+            );
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(encoded.rejected.len() as u64, Ordering::Relaxed);
+
+        let mut written = 0u64;
+        let mut last_error = None;
+        for chunk in &encoded.chunks {
+            match self
+                .write_chunk(client, chunk, topic_metadata, &messages_metadata)
+                .await
+            {
+                Ok(rows) => written += rows,
+                Err(e) => last_error = Some(e),
+            }
+        }
+        self.counters
+            .rows_written
+            .fetch_add(written, Ordering::Relaxed);
+
+        let summary = format!(

Review Comment:
   nit: `consume` builds the summary string before the log level test, so every batch allocates it even when both levels are off. Pass the fields to `info!` and `debug!` directly.



##########
core/connectors/sinks/bigquery_sink/tests/bigquery_sink.rs:
##########
@@ -0,0 +1,424 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   warning: No test under `core/integration/tests/connectors/` runs this sink against a real backend. The repository requires an integration test for every sink that talks to an external service. Add a BigQuery fixture and test, or get maintainer sign-off for the gap.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, the run is decoded again row by
+//! row so that only the offending rows are dropped.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, TimestampMicrosecondArray,
+};
+use arrow::compute::{cast, concat_batches};
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, kept.len(), |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(to_hex(&bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    row_count: usize,
+    mut on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() {
+        return Ok((None, (0..row_count).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    if let Ok(batch) = decode(&schema, rows) {
+        return Ok((Some(to_target(batch, &target)?), (0..row_count).collect()));
+    }
+
+    let mut batches = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in rows.iter().enumerate() {
+        match decode(&schema, std::slice::from_ref(row)) {

Review Comment:
   warning: When one row fails, `decode_rows` decodes every row on its own and `concat_batches` copies every survivor again. Bisect the failing run to find the bad rows, then decode only those.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, the run is decoded again row by
+//! row so that only the offending rows are dropped.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, TimestampMicrosecondArray,
+};
+use arrow::compute::{cast, concat_batches};
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, kept.len(), |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(to_hex(&bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    row_count: usize,
+    mut on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() {
+        return Ok((None, (0..row_count).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    if let Ok(batch) = decode(&schema, rows) {
+        return Ok((Some(to_target(batch, &target)?), (0..row_count).collect()));
+    }
+
+    let mut batches = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in rows.iter().enumerate() {
+        match decode(&schema, std::slice::from_ref(row)) {
+            Ok(batch) => {
+                batches.push(batch);
+                survivors.push(index);
+            }
+            Err(e) => on_reject(index, format!("row does not match the table schema: {e}")),
+        }
+    }
+    if batches.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = concat_batches(&schema, &batches)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    decoder.serialize(rows)?;
+    decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut values = Vec::with_capacity(messages.len());
+            for message in messages {
+                match message.payload.try_to_bytes() {
+                    Ok(bytes) => {
+                        values.push(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(BinaryArray::from_iter_values(values))
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_owned_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => Arc::new(StringArray::from_iter_values(
+                messages.iter().map(|m| m.id.to_string()),
+            )),
+            MetaColumn::Headers => {
+                let required = !field.is_nullable();
+                Arc::new(StringArray::from_iter(
+                    messages.iter().map(|m| headers_json(m, required)),
+                ))
+            }
+        };
+        fields.push(field.clone());
+        columns.push(column);
+    }
+    Ok(())
+}
+
+/// Headers as a JSON object. Text values are written as strings, binary
+/// values as `{"data": <base64>, "iggy_header_encoding": "base64"}`, the same
+/// shape `http_sink` uses.
+fn headers_json(message: &ConsumedMessage, required: bool) -> Option<String> {
+    let headers = message.headers.as_ref().filter(|h| !h.is_empty());
+    let Some(headers) = headers else {
+        return required.then(|| "{}".to_owned());
+    };
+    let mut object = serde_json::Map::with_capacity(headers.len());
+    for (key, value) in headers {
+        let encoded = match value.as_raw() {
+            Ok(raw) => serde_json::json!({
+                "data": BASE64.encode(raw),
+                "iggy_header_encoding": HEADER_ENCODING_BASE64,
+            }),
+            Err(_) => serde_json::Value::String(value.to_string_value()),
+        };
+        object.insert(key.to_string_value(), encoded);
+    }
+    Some(serde_json::Value::Object(object).to_string())
+}
+
+// ─── Request sizing ──────────────────────────────────────────────────────────
+
+/// Encode `batch` and halve it until every piece fits `max_bytes`. A single
+/// row that does not fit is rejected.
+fn split_into_chunks(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<(), ArrowError> {
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    let size = schema_bytes.len() + batch_bytes.len();
+    if size <= max_bytes {
+        chunks.push(Chunk {
+            batch,
+            offsets,
+            schema_bytes,
+            batch_bytes,
+        });
+        return Ok(());
+    }
+    if batch.num_rows() == 1 {
+        rejected.push(Rejected {
+            offset: offsets[0],
+            reason: format!("row is {size} bytes encoded, above max_request_bytes ({max_bytes})"),
+        });
+        return Ok(());
+    }
+    let half = batch.num_rows() / 2;
+    let (left_offsets, right_offsets) = offsets.split_at(half);
+    split_into_chunks(
+        batch.slice(0, half),
+        left_offsets.to_vec(),
+        max_bytes,
+        chunks,
+        rejected,
+    )?;
+    split_into_chunks(
+        batch.slice(half, batch.num_rows() - half),
+        right_offsets.to_vec(),
+        max_bytes,
+        chunks,
+        rejected,
+    )
+}
+
+/// Serialize the schema and the batch as Arrow IPC stream messages, the
+/// format `AppendRowsRequest.arrow_rows` carries.
+fn encode_ipc(batch: &RecordBatch) -> Result<(Vec<u8>, Vec<u8>), ArrowError> {
+    let options = IpcWriteOptions::default();
+    let generator = IpcDataGenerator::default();
+    let mut tracker = DictionaryTracker::new(true);
+    let mut compression = CompressionContext::default();
+
+    let schema = generator.schema_to_bytes_with_dictionary_tracker(
+        batch.schema_ref(),
+        &mut tracker,
+        &options,
+    );
+    let mut schema_bytes = Vec::new();
+    write_message(&mut schema_bytes, schema, &options)?;
+
+    let (dictionaries, encoded) =
+        generator.encode(batch, &mut tracker, &options, &mut compression)?;
+    let mut batch_bytes = Vec::new();
+    for dictionary in dictionaries {
+        write_message(&mut batch_bytes, dictionary, &options)?;
+    }
+    write_message(&mut batch_bytes, encoded, &options)?;
+    Ok((schema_bytes, batch_bytes))
+}
+
+// ─── Helpers ─────────────────────────────────────────────────────────────────
+
+fn payload_kind(payload: &Payload) -> &'static str {
+    match payload {
+        Payload::Json(_) => "JSON",
+        Payload::Raw(_) => "raw",
+        Payload::Text(_) => "text",
+        Payload::Proto(_) => "proto",
+        Payload::FlatBuffer(_) => "FlatBuffer",
+        Payload::Avro(_) => "Avro",
+    }
+}
+
+fn empty_array() -> OwnedValue {
+    OwnedValue::Array(Box::default())
+}
+
+fn to_hex(bytes: &[u8]) -> String {

Review Comment:
   simplification: `to_hex` hand-rolls what `hex::encode` does, and the workspace already depends on `hex`. Add `hex = { workspace = true }` to the crate, then delete `to_hex` and its `std::fmt::Write` import.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "kparisa (via GitHub)" <gi...@apache.org>.
kparisa commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4175754498


##########
core/connectors/sinks/bigquery_sink/src/error.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Failure classification for BigQuery calls.
+//!
+//! Only transient failures are retried. Retrying a permanent one (bad
+//! schema, missing permission) cannot succeed and only delays the error.
+
+use crate::schema::SchemaError;
+use gcloud_gax::grpc::{Code, Status};
+use iggy_connector_sdk::Error;
+use std::fmt;
+
+/// An `AppendRows` call that did not append. Row-level errors are not part
+/// of this type: they are an expected outcome the caller handles by
+/// dropping the reported rows.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum AppendError {
+    /// The call failed, or the response carried an error status.
+    Rpc { code: Code, message: String },
+    /// The response stream closed without a response.
+    NoResponse,
+}
+
+/// A `tables.get` call that did not return a usable schema.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum TableError {
+    Http { status: u16, body: String },
+    Transport(String),
+    Token(String),
+    Schema(SchemaError),
+}
+
+impl AppendError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            AppendError::Rpc { code, .. } => is_retryable_code(*code),
+            // The append may or may not have landed. Retrying risks a
+            // duplicate, which the delivery guarantees allow.
+            AppendError::NoResponse => true,
+        }
+    }
+}
+
+impl TableError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            TableError::Http { status, .. } => *status == 429 || *status >= 500,
+            TableError::Transport(_) => true,
+            TableError::Token(_) | TableError::Schema(_) => false,

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4394:
URL: https://github.com/apache/iggy/pull/4394#issuecomment-5967979830

   ## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4394?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 87.96%. Comparing base ([`7c10d8a`](https://app.codecov.io/gh/apache/iggy/commit/7c10d8adbe846143234307c00efabec2f77ddb43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8e518f8`](https://app.codecov.io/gh/apache/iggy/commit/8e518f86c441a3dc6d745ab57e713ffc7c8fd172?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #4394      +/-   ##
   ============================================
   + Coverage     87.86%   87.96%   +0.09%     
   + Complexity     1579     1578       -1     
   ============================================
     Files          1290     1290              
     Lines        230145   230145              
     Branches     193482   193481       -1     
   ============================================
   + Hits         202228   202445     +217     
   + Misses        23195    22977     -218     
   - Partials       4722     4723       +1     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.06% <ø> (ø)` | |
   | [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.72% <ø> (-0.02%)` | :arrow_down: |
   | [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.70% <ø> (+0.01%)` | :arrow_up: |
   | [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.24% <ø> (ø)` | |
   | [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.67% <ø> (ø)` | |
   | [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.57% <ø> (+1.68%)` | :arrow_up: |
   | [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4394/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.29% <ø> (+0.02%)` | :arrow_up: |
   [see 4 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4394/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4212463823


##########
core/connectors/sinks/bigquery_sink/src/schema.rs:
##########
@@ -0,0 +1,769 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! BigQuery table schema model and its mapping to Arrow.
+//!
+//! The schema comes from the `tables.get` REST call as JSON. It is parsed
+//! with local serde types rather than a client library model so that every
+//! BigQuery type (GEOGRAPHY, RANGE, ...) deserializes, and the unsupported
+//! ones can be rejected with a clear message in `open()`.
+//!
+//! Arrow types follow the Storage Write API Arrow mapping:
+//!
+//! | BigQuery   | Arrow                          |
+//! |------------|--------------------------------|
+//! | STRING     | Utf8                           |
+//! | BYTES      | Binary                         |
+//! | INT64      | Int64                          |
+//! | FLOAT64    | Float64                        |
+//! | BOOL       | Boolean                        |
+//! | NUMERIC    | Decimal128(38, 9)              |
+//! | BIGNUMERIC | Decimal256(76, 38)             |
+//! | TIMESTAMP  | Timestamp(Microsecond, "UTC")  |
+//! | DATETIME   | Timestamp(Microsecond, none)   |
+//! | DATE       | Date32                         |
+//! | TIME       | Time64(Microsecond)            |
+//! | GEOGRAPHY  | Utf8 (WKT)                     |
+//! | JSON       | Utf8 (JSON text)               |
+//! | RECORD     | Struct                         |
+//! | REPEATED   | List                           |
+
+use crate::{Settings, WriteMode};
+use arrow::datatypes::{DataType, Field, FieldRef, Fields, TimeUnit};
+use serde::Deserialize;
+use std::collections::HashMap;
+use std::fmt;
+use std::sync::Arc;
+
+pub(crate) const UTC: &str = "UTC";
+const LIST_ITEM: &str = "item";
+
+/// One table column, recursively for RECORD types.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct Column {
+    pub name: String,
+    pub bq_type: BqType,
+    pub mode: Mode,
+    pub has_default: bool,
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) enum BqType {
+    String,
+    Bytes,
+    Int64,
+    Float64,
+    Bool,
+    Numeric,
+    BigNumeric,
+    Timestamp,
+    Datetime,
+    Date,
+    Time,
+    Geography,
+    Json,
+    Record(Vec<Column>),
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum Mode {
+    Nullable,
+    Required,
+    Repeated,
+}
+
+/// Iggy metadata written next to the payload.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum MetaColumn {
+    Stream,
+    Topic,
+    PartitionId,
+    Offset,
+    Timestamp,
+    Id,
+    Headers,
+}
+
+/// What the `raw` mode payload column holds.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum RawKind {
+    Json,
+    String,
+    Bytes,
+}
+
+/// The validated mapping from messages to the target table, built once in
+/// `open()`.
+#[derive(Debug)]
+pub(crate) struct TableLayout {
+    /// Payload columns in table order. Empty in `raw` mode.
+    pub columns: Vec<Column>,
+    /// Arrow field for each entry of `columns`, same index.
+    pub fields: Vec<FieldRef>,
+    /// Column name to index into `columns`.
+    pub index: HashMap<String, usize>,
+    /// Enabled metadata columns with their Arrow fields.
+    pub metadata: Vec<(MetaColumn, FieldRef)>,
+    /// `raw` mode payload column.
+    pub raw: Option<(RawKind, FieldRef)>,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum SchemaError {
+    Malformed(String),
+    EmptySchema,
+    UnsupportedType {
+        column: String,
+        bq_type: String,
+    },
+    MissingColumn {
+        column: String,
+        hint: String,
+    },
+    WrongType {
+        column: String,
+        expected: String,
+        actual: String,
+    },
+    UnfillableColumn(String),
+}
+
+impl fmt::Display for SchemaError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            SchemaError::Malformed(reason) => write!(f, "malformed table schema: {reason}"),
+            SchemaError::EmptySchema => write!(f, "table has no columns"),
+            SchemaError::UnsupportedType { column, bq_type } => write!(
+                f,
+                "column '{column}' has type {bq_type}, which the BigQuery sink does not support"
+            ),
+            SchemaError::MissingColumn { column, hint } => {
+                write!(f, "column '{column}' is missing from the table. {hint}")
+            }
+            SchemaError::WrongType {
+                column,
+                expected,
+                actual,
+            } => write!(f, "column '{column}' must be {expected}, found {actual}"),
+            SchemaError::UnfillableColumn(column) => write!(
+                f,
+                "column '{column}' is REQUIRED without a default value and raw mode never writes it"
+            ),
+        }
+    }
+}
+
+impl MetaColumn {
+    pub(crate) const ALL: [MetaColumn; 7] = [
+        MetaColumn::Stream,
+        MetaColumn::Topic,
+        MetaColumn::PartitionId,
+        MetaColumn::Offset,
+        MetaColumn::Timestamp,
+        MetaColumn::Id,
+        MetaColumn::Headers,
+    ];
+
+    pub(crate) fn name(self) -> &'static str {
+        match self {
+            MetaColumn::Stream => "iggy_stream",
+            MetaColumn::Topic => "iggy_topic",
+            MetaColumn::PartitionId => "iggy_partition_id",
+            MetaColumn::Offset => "iggy_offset",
+            MetaColumn::Timestamp => "iggy_timestamp",
+            MetaColumn::Id => "iggy_id",
+            MetaColumn::Headers => "iggy_headers",
+        }
+    }
+
+    fn ddl_type(self) -> &'static str {
+        match self {
+            MetaColumn::Stream | MetaColumn::Topic | MetaColumn::Id => "STRING",
+            MetaColumn::PartitionId | MetaColumn::Offset => "INT64",
+            MetaColumn::Timestamp => "TIMESTAMP",
+            MetaColumn::Headers => "JSON",
+        }
+    }
+
+    fn accepts(self, bq_type: &BqType) -> bool {
+        match self {
+            MetaColumn::Stream | MetaColumn::Topic | MetaColumn::Id => {
+                matches!(bq_type, BqType::String)
+            }
+            MetaColumn::PartitionId | MetaColumn::Offset => matches!(bq_type, BqType::Int64),
+            MetaColumn::Timestamp => matches!(bq_type, BqType::Timestamp),
+            MetaColumn::Headers => matches!(bq_type, BqType::Json | BqType::String),
+        }
+    }
+
+    fn enabled(self, settings: &Settings) -> bool {
+        match self {
+            MetaColumn::Headers => settings.include_headers,
+            _ => settings.include_metadata,
+        }
+    }
+}
+
+/// Parse the body of a `tables.get` response.
+pub(crate) fn parse_table_schema(body: &[u8]) -> Result<Vec<Column>, SchemaError> {
+    let resource: TableResource =
+        serde_json::from_slice(body).map_err(|e| SchemaError::Malformed(e.to_string()))?;
+    let fields = resource
+        .schema
+        .map(|schema| schema.fields)
+        .unwrap_or_default();
+    if fields.is_empty() {
+        return Err(SchemaError::EmptySchema);
+    }
+    fields.iter().map(Column::try_from).collect()
+}
+
+impl TableLayout {
+    pub(crate) fn build(columns: Vec<Column>, settings: &Settings) -> Result<Self, SchemaError> {
+        let mut by_name: HashMap<&str, &Column> =
+            columns.iter().map(|c| (c.name.as_str(), c)).collect();
+
+        let mut metadata = Vec::new();
+        for meta in MetaColumn::ALL {
+            if !meta.enabled(settings) {
+                continue;
+            }
+            let column = by_name
+                .remove(meta.name())
+                .ok_or_else(|| SchemaError::MissingColumn {
+                    column: meta.name().to_owned(),
+                    hint: format!(
+                        "Add `{} {}` to the table or disable it in the connector config.",
+                        meta.name(),
+                        meta.ddl_type()
+                    ),
+                })?;
+            if column.mode == Mode::Repeated || !meta.accepts(&column.bq_type) {
+                return Err(SchemaError::WrongType {
+                    column: column.name.clone(),
+                    expected: meta.ddl_type().to_owned(),
+                    actual: column.type_label(),
+                });
+            }
+            metadata.push((meta, Arc::new(meta.arrow_field(column.mode))));
+        }
+
+        match settings.mode {
+            WriteMode::Raw => Self::build_raw(&columns, by_name, metadata, settings),
+            WriteMode::Mapped => Ok(Self::build_mapped(&columns, &metadata)),
+        }
+    }
+
+    fn build_raw(
+        columns: &[Column],
+        mut remaining: HashMap<&str, &Column>,
+        metadata: Vec<(MetaColumn, FieldRef)>,
+        settings: &Settings,
+    ) -> Result<Self, SchemaError> {
+        let name = settings.payload_column.as_str();
+        let column = remaining
+            .remove(name)
+            .ok_or_else(|| SchemaError::MissingColumn {
+                column: name.to_owned(),
+                hint: "Add a JSON, STRING or BYTES column for the payload or set payload_column."
+                    .to_owned(),
+            })?;
+        let kind = match (&column.bq_type, column.mode) {
+            (BqType::Json, Mode::Nullable | Mode::Required) => RawKind::Json,
+            (BqType::String, Mode::Nullable | Mode::Required) => RawKind::String,
+            (BqType::Bytes, Mode::Nullable | Mode::Required) => RawKind::Bytes,
+            _ => {
+                return Err(SchemaError::WrongType {
+                    column: column.name.clone(),
+                    expected: "JSON, STRING or BYTES".to_owned(),
+                    actual: column.type_label(),
+                });
+            }
+        };
+
+        // Every other column is left out of the writer schema, so BigQuery
+        // fills it from its default. A REQUIRED column without one would
+        // reject every row.
+        if let Some(unfillable) = columns.iter().find(|c| {
+            remaining.contains_key(c.name.as_str()) && c.mode == Mode::Required && !c.has_default
+        }) {
+            return Err(SchemaError::UnfillableColumn(unfillable.name.clone()));
+        }
+
+        let field = Arc::new(column.arrow_field());
+        Ok(TableLayout {
+            columns: Vec::new(),
+            fields: Vec::new(),
+            index: HashMap::new(),
+            metadata,
+            raw: Some((kind, field)),
+        })
+    }
+
+    fn build_mapped(columns: &[Column], metadata: &[(MetaColumn, FieldRef)]) -> Self {
+        let payload_columns: Vec<Column> = columns
+            .iter()
+            .filter(|c| !metadata.iter().any(|(meta, _)| meta.name() == c.name))
+            .cloned()
+            .collect();
+        let fields = payload_columns
+            .iter()
+            .map(|c| Arc::new(c.arrow_field()))
+            .collect();
+        let index = payload_columns
+            .iter()
+            .enumerate()
+            .map(|(i, c)| (c.name.clone(), i))
+            .collect();
+        TableLayout {
+            columns: payload_columns,
+            fields,
+            index,
+            metadata: metadata.to_vec(),
+            raw: None,
+        }
+    }
+}
+
+impl Column {
+    /// Arrow field as the Storage Write API expects it. REQUIRED maps to a
+    /// non-nullable field. REPEATED maps to a list whose items are
+    /// non-nullable, because BigQuery arrays cannot hold NULL.
+    pub(crate) fn arrow_field(&self) -> Field {
+        let element = self.bq_type.arrow_type();
+        match self.mode {
+            Mode::Nullable => Field::new(&self.name, element, true),
+            Mode::Required => Field::new(&self.name, element, false),
+            Mode::Repeated => Field::new(
+                &self.name,
+                DataType::List(Arc::new(Field::new(LIST_ITEM, element, false))),
+                true,
+            ),
+        }
+    }
+
+    fn type_label(&self) -> String {
+        let base = self.bq_type.label();
+        match self.mode {
+            Mode::Repeated => format!("REPEATED {base}"),
+            _ => base.to_owned(),
+        }
+    }
+}
+
+impl BqType {
+    fn arrow_type(&self) -> DataType {
+        match self {
+            BqType::String | BqType::Geography | BqType::Json => DataType::Utf8,
+            BqType::Bytes => DataType::Binary,
+            BqType::Int64 => DataType::Int64,
+            BqType::Float64 => DataType::Float64,
+            BqType::Bool => DataType::Boolean,
+            BqType::Numeric => DataType::Decimal128(38, 9),

Review Comment:
    NUMERIC/BIGNUMERIC hardcoded to Decimal128(38,9)/Decimal256(76,38) regardless of the table's real declared scale; RestField never
       deserializes BigQuery's actual precision/scale from tables.get (zero matches, grepped). arrow-cast 58.4.0 (the actually-locked version, verified against parse.rs:914-924) silently truncates fractional digits
   
   



##########
core/connectors/sinks/bigquery_sink/src/sink.rs:
##########
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::client::{AppendOutcome, BigQueryClient};
+use crate::encode::{self, Chunk, RunContext};
+use crate::error::{AppendError, TableError};
+use crate::schema::{TableLayout, parse_table_schema};
+use crate::{BigQuerySink, OpenState};
+use async_trait::async_trait;
+use iggy_connector_sdk::retry::{RetryPolicy, retry_async};
+use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata};
+use std::sync::atomic::Ordering;
+use tracing::{debug, error, info, warn};
+
+#[async_trait]
+impl Sink for BigQuerySink {
+    async fn open(&mut self) -> Result<(), Error> {
+        info!(
+            "Opening BigQuery sink connector ID: {} for table {} (mode: {:?})",
+            self.id, self.target, self.settings.mode
+        );
+        // The SDK turns a failed open into a bare status code, so this is the
+        // only place the reason reaches the runtime log.
+        self.try_open().await.inspect_err(|e| {
+            error!(
+                "Failed to open BigQuery sink connector ID: {} for table {}: {e}",
+                self.id, self.target
+            );
+        })
+    }
+
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        if messages.is_empty() {
+            return Ok(());
+        }
+        let Some(state) = &self.state else {
+            return Err(Error::InitError("BigQuery sink is not open".into()));
+        };
+        let received = messages.len();
+        debug!(
+            "BigQuery sink ID: {} received {received} messages from {}/{} partition {} current_offset {}",
+            self.id,
+            topic_metadata.stream,
+            topic_metadata.topic,
+            messages_metadata.partition_id,
+            messages_metadata.current_offset
+        );
+
+        let ctx = RunContext {
+            layout: &state.layout,
+            topic: topic_metadata,
+            messages: &messages_metadata,
+            max_request_bytes: self.settings.max_request_bytes,
+        };
+        let encoded = encode::encode(&ctx, messages).map_err(|e| {
+            self.counters
+                .rows_failed
+                .fetch_add(received as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} cannot encode {received} messages from {}/{} partition {}: {e}",
+                self.id, topic_metadata.stream, topic_metadata.topic, messages_metadata.partition_id
+            );
+            Error::Serialization(e.to_string())
+        })?;
+
+        for rejected in &encoded.rejected {
+            warn!(
+                "BigQuery sink ID: {} dropped message stream={} topic={} partition={} offset={}: {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                rejected.offset,
+                rejected.reason
+            );
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(encoded.rejected.len() as u64, Ordering::Relaxed);
+
+        let mut written = 0u64;
+        let mut last_error = None;
+        for chunk in &encoded.chunks {
+            match self
+                .write_chunk(&state.client, chunk, topic_metadata, &messages_metadata)
+                .await
+            {
+                Ok(rows) => written += rows,
+                Err(e) => last_error = Some(e),
+            }
+        }
+        self.counters
+            .rows_written
+            .fetch_add(written, Ordering::Relaxed);
+
+        if self.settings.verbose {
+            info!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        } else {
+            debug!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        }
+
+        match last_error {
+            Some(e) => Err(e),
+            None => Ok(()),
+        }
+    }
+
+    async fn close(&mut self) -> Result<(), Error> {
+        self.state = None;
+        info!(
+            "Closed BigQuery sink connector ID: {}, rows written: {}, rows rejected: {}, rows failed: {}",
+            self.id,
+            self.counters.rows_written.load(Ordering::Relaxed),
+            self.counters.rows_rejected.load(Ordering::Relaxed),
+            self.counters.rows_failed.load(Ordering::Relaxed)
+        );
+        Ok(())
+    }
+}
+
+impl BigQuerySink {
+    async fn try_open(&mut self) -> Result<(), Error> {
+        self.config.validate()?;
+
+        let mut client = BigQueryClient::connect(&self.config, &self.settings).await?;
+        let policy = self.retry_policy();
+
+        let context = format!("BigQuery sink ID: {} tables.get", self.id);
+        let body = retry_async(policy, &context, TableError::is_retryable, || {
+            client.fetch_table()
+        })
+        .await
+        .map_err(|failure| Error::InitError(format!("cannot read the table schema: {failure}")))?;
+
+        let columns = parse_table_schema(&body).map_err(|e| Error::from(TableError::Schema(e)))?;
+        let layout = TableLayout::build(columns, &self.settings)
+            .map_err(|e| Error::from(TableError::Schema(e)))?;
+
+        let context = format!("BigQuery sink ID: {} write stream setup", self.id);
+        let stream = retry_async(policy, &context, AppendError::is_retryable, || {
+            client.resolve_write_stream()
+        })
+        .await
+        .map_err(|failure| Error::InitError(format!("cannot open the write stream: {failure}")))?;
+        client.set_write_stream(stream);
+
+        info!(
+            "Opened BigQuery sink connector ID: {} for table {}: {} payload column(s), {} metadata column(s)",
+            self.id,
+            self.target,
+            layout.raw.as_ref().map_or(layout.columns.len(), |_| 1),
+            layout.metadata.len()
+        );
+        self.state = Some(OpenState { client, layout });
+        Ok(())
+    }
+
+    fn retry_policy(&self) -> RetryPolicy {
+        RetryPolicy {
+            max_attempts: self.settings.max_retries,
+            base_delay: self.settings.retry_delay,
+            max_delay: self.settings.max_retry_delay,
+        }
+    }
+
+    /// Append one chunk. When BigQuery reports row errors, nothing was
+    /// appended: drop the reported rows and append the rest once more.
+    /// Returns the number of rows written.
+    async fn write_chunk(
+        &self,
+        client: &BigQueryClient,
+        chunk: &Chunk,
+        topic: &TopicMetadata,
+        messages: &MessagesMetadata,
+    ) -> Result<u64, Error> {
+        let row_errors = match self
+            .append_with_retry(client, chunk, topic, messages)
+            .await?
+        {
+            AppendOutcome::Appended => return Ok(chunk.offsets.len() as u64),
+            AppendOutcome::RowErrors(row_errors) => row_errors,
+        };
+
+        let mut dropped = Vec::with_capacity(row_errors.len());
+        for (row, reason) in row_errors {
+            let offset = chunk.offsets.get(row).copied();
+            warn!(
+                "BigQuery sink ID: {} dropped row rejected by BigQuery stream={} topic={} partition={} offset={}: {reason}",
+                self.id,
+                topic.stream,
+                topic.topic,
+                messages.partition_id,
+                offset.map_or_else(|| format!("unknown (row {row})"), |o| o.to_string())
+            );
+            dropped.push(row);
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(dropped.len() as u64, Ordering::Relaxed);
+
+        let Some(retry) = encode::without_rows(chunk, &dropped)
+            .map_err(|e| Error::Serialization(e.to_string()))?
+        else {
+            return Ok(0);
+        };
+        match self
+            .append_with_retry(client, &retry, topic, messages)
+            .await?
+        {
+            AppendOutcome::Appended => Ok(retry.offsets.len() as u64),
+            AppendOutcome::RowErrors(again) => {
+                self.counters
+                    .rows_failed
+                    .fetch_add(retry.offsets.len() as u64, Ordering::Relaxed);
+                error!(
+                    "BigQuery sink ID: {} dropped {} rows (offsets {}..={}) of {}/{} partition {}: BigQuery reported {} more row error(s) after the bad rows were removed",
+                    self.id,
+                    retry.offsets.len(),
+                    retry.offsets.first().copied().unwrap_or_default(),
+                    retry.offsets.last().copied().unwrap_or_default(),
+                    topic.stream,
+                    topic.topic,
+                    messages.partition_id,
+                    again.len()
+                );
+                Err(Error::PermanentHttpError(format!(
+                    "{} row error(s) persisted after removing rejected rows",
+                    again.len()
+                )))
+            }
+        }
+    }
+
+    async fn append_with_retry(

Review Comment:
   core/connectors/runtime/src/sink.rs:475-525 × core/connectors/runtime/src/manager/sink.rs:103-114 — exhausted-but-still-retryable AppendRows   failures touch no counter (error! log only). Runtime hardcodes AutoCommit::When(AutoCommitWhen::PollingMessages) (commits the Iggy offset before consume() runs), consume()'s Err ends the whole spawned task,   and set_error only flips ConnectorStatus::Error with no restart/redelivery anywhere. Net effect: silent, permanent, unmetered row loss, plus the entire partition stops being consumed until an operator notices
       and restarts. **Fix**: count these rows, make the failure an alertable metric not a log line, rewrite README.md:155-172 to lead with permanent-loss-and-halt, not just "may duplicate on retry."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on PR #4394:
URL: https://github.com/apache/iggy/pull/4394#issuecomment-5970907117

   /skill team-review-slim


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "kparisa (via GitHub)" <gi...@apache.org>.
kparisa commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4221504972


##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);

Review Comment:
   yes, its expected to set the values NULL when not present and reject the specific entry when it doesn't carry required fields. it will log like this
   --
   2026-10-08T13:54:06.544169Z  WARN connector: connector_target="iggy_connector_bigquery_sink::sink" BigQuery sink ID: 1 dropped message stream=demo_stream topic=demo_topic partition=0 offset=14: row does not match the table schema: Json error: whilst decoding field 'user_id': failed to parse "bad" as Int64



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4212379611


##########
core/connectors/sinks/bigquery_sink/src/sink.rs:
##########
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::client::{AppendOutcome, BigQueryClient};
+use crate::encode::{self, Chunk, RunContext};
+use crate::error::{AppendError, TableError};
+use crate::schema::{TableLayout, parse_table_schema};
+use crate::{BigQuerySink, OpenState};
+use async_trait::async_trait;
+use iggy_connector_sdk::retry::{RetryPolicy, retry_async};
+use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata};
+use std::sync::atomic::Ordering;
+use tracing::{debug, error, info, warn};
+
+#[async_trait]
+impl Sink for BigQuerySink {
+    async fn open(&mut self) -> Result<(), Error> {
+        info!(
+            "Opening BigQuery sink connector ID: {} for table {} (mode: {:?})",
+            self.id, self.target, self.settings.mode
+        );
+        // The SDK turns a failed open into a bare status code, so this is the
+        // only place the reason reaches the runtime log.
+        self.try_open().await.inspect_err(|e| {
+            error!(
+                "Failed to open BigQuery sink connector ID: {} for table {}: {e}",
+                self.id, self.target
+            );
+        })
+    }
+
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        if messages.is_empty() {
+            return Ok(());
+        }
+        let Some(state) = &self.state else {
+            return Err(Error::InitError("BigQuery sink is not open".into()));
+        };
+        let received = messages.len();
+        debug!(
+            "BigQuery sink ID: {} received {received} messages from {}/{} partition {} current_offset {}",
+            self.id,
+            topic_metadata.stream,
+            topic_metadata.topic,
+            messages_metadata.partition_id,
+            messages_metadata.current_offset
+        );
+
+        let ctx = RunContext {
+            layout: &state.layout,
+            topic: topic_metadata,
+            messages: &messages_metadata,
+            max_request_bytes: self.settings.max_request_bytes,
+        };
+        let encoded = encode::encode(&ctx, messages).map_err(|e| {
+            self.counters
+                .rows_failed
+                .fetch_add(received as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} cannot encode {received} messages from {}/{} partition {}: {e}",
+                self.id, topic_metadata.stream, topic_metadata.topic, messages_metadata.partition_id
+            );
+            Error::Serialization(e.to_string())
+        })?;
+
+        for rejected in &encoded.rejected {
+            warn!(
+                "BigQuery sink ID: {} dropped message stream={} topic={} partition={} offset={}: {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                rejected.offset,
+                rejected.reason
+            );
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(encoded.rejected.len() as u64, Ordering::Relaxed);
+
+        let mut written = 0u64;
+        let mut last_error = None;
+        for chunk in &encoded.chunks {
+            match self
+                .write_chunk(&state.client, chunk, topic_metadata, &messages_metadata)
+                .await
+            {
+                Ok(rows) => written += rows,
+                Err(e) => last_error = Some(e),
+            }
+        }
+        self.counters
+            .rows_written
+            .fetch_add(written, Ordering::Relaxed);
+
+        if self.settings.verbose {
+            info!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        } else {
+            debug!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        }
+
+        match last_error {
+            Some(e) => Err(e),
+            None => Ok(()),
+        }
+    }
+
+    async fn close(&mut self) -> Result<(), Error> {
+        self.state = None;
+        info!(
+            "Closed BigQuery sink connector ID: {}, rows written: {}, rows rejected: {}, rows failed: {}",
+            self.id,
+            self.counters.rows_written.load(Ordering::Relaxed),
+            self.counters.rows_rejected.load(Ordering::Relaxed),
+            self.counters.rows_failed.load(Ordering::Relaxed)
+        );
+        Ok(())
+    }
+}
+
+impl BigQuerySink {
+    async fn try_open(&mut self) -> Result<(), Error> {
+        self.config.validate()?;
+
+        let mut client = BigQueryClient::connect(&self.config, &self.settings).await?;
+        let policy = self.retry_policy();
+
+        let context = format!("BigQuery sink ID: {} tables.get", self.id);
+        let body = retry_async(policy, &context, TableError::is_retryable, || {
+            client.fetch_table()
+        })
+        .await
+        .map_err(|failure| Error::InitError(format!("cannot read the table schema: {failure}")))?;
+
+        let columns = parse_table_schema(&body).map_err(|e| Error::from(TableError::Schema(e)))?;
+        let layout = TableLayout::build(columns, &self.settings)
+            .map_err(|e| Error::from(TableError::Schema(e)))?;
+
+        let context = format!("BigQuery sink ID: {} write stream setup", self.id);
+        let stream = retry_async(policy, &context, AppendError::is_retryable, || {

Review Comment:
   warning: gax already retries `create_write_stream` on `Unavailable` (21 calls, backoff capped at 60s), and `retry_async` runs that 3 times, so a Storage Write API outage blocks `open()` and runtime startup for tens of minutes. wrap the `retry_async` call in one `tokio::time::timeout`.



##########
core/connectors/sinks/bigquery_sink/src/error.rs:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Failure classification for BigQuery calls.
+//!
+//! Only transient failures are retried. Retrying a permanent one (bad
+//! schema, missing permission) cannot succeed and only delays the error.
+
+use crate::schema::SchemaError;
+use gcloud_gax::grpc::{Code, Status};
+use iggy_connector_sdk::Error;
+use std::fmt;
+
+/// An `AppendRows` call that did not append. Row-level errors are not part
+/// of this type: they are an expected outcome the caller handles by
+/// dropping the reported rows.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum AppendError {
+    /// The call failed, or the response carried an error status.
+    Rpc { code: Code, message: String },
+    /// The response stream closed without a response.
+    NoResponse,
+}
+
+/// A `tables.get` call that did not return a usable schema.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum TableError {
+    Http { status: u16, body: String },
+    Transport(String),
+    Token(String),
+    Schema(SchemaError),
+}
+
+impl AppendError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            AppendError::Rpc { code, .. } => is_retryable_code(*code),
+            // The append may or may not have landed. Retrying risks a
+            // duplicate, which the delivery guarantees allow.
+            AppendError::NoResponse => true,
+        }
+    }
+}
+
+impl TableError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            TableError::Http { status, .. } => *status == 429 || *status >= 500,
+            TableError::Transport(_) | TableError::Token(_) => true,
+            TableError::Schema(_) => false,
+        }
+    }
+}
+
+/// gRPC codes BigQuery documents as safe to retry for `AppendRows`.
+pub(crate) fn is_retryable_code(code: Code) -> bool {

Review Comment:
   critical: same problem `Cancelled` had - `Unknown` (connection reset) and `Unauthenticated` (failed token refresh) are permanent here, so one network blip drops the chunk after its offset is committed. retry both, together with an overall timeout in `try_open`.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);

Review Comment:
   warning: the writer schema covers every key in the batch, so if one row sets a defaulted column, rows that skip it get NULL or are rejected, not the default. group rows by defaulted keys they set, one schema per group.



##########
core/connectors/sinks/bigquery_sink/README.md:
##########
@@ -0,0 +1,264 @@
+# BigQuery Sink Connector
+
+The BigQuery sink connector consumes messages from Iggy topics and writes them into a Google BigQuery table through the [Storage Write API](https://cloud.google.com/bigquery/docs/write-api) `_default` stream, with rows encoded as Arrow record batches.
+
+## Features
+
+- **Storage Write API**: gRPC `AppendRows` on the table's `_default` stream. No legacy `insertAll`.
+- **Two modes**: `mapped` maps top-level JSON fields to table columns by name. `raw` stores the whole payload in one JSON, STRING or BYTES column.
+- **Schema validation at startup**: the table schema is read with `tables.get` and checked before any data flows. Unsupported column types fail `open()`.
+- **Iggy metadata columns**: stream, topic, partition, offset, timestamp and message id, plus optional headers, for lineage and deduplication.
+- **Per-row error handling**: a row that cannot be written is dropped and logged with its offset. The rest of the batch is still written.
+- **Request sizing**: batches are split below the 10 MB `AppendRows` limit.
+- **Retries**: transient gRPC failures are retried with jittered exponential backoff.
+
+## Configuration
+
+```toml
+type = "sink"
+key = "bigquery"
+enabled = true
+version = 0
+name = "BigQuery sink"
+path = "target/release/libiggy_connector_bigquery_sink"
+
+[[streams]]
+stream = "example_stream"
+topics = ["example_topic"]
+schema = "json"
+batch_length = 1000
+poll_interval = "5ms"
+consumer_group = "bigquery_sink_connector"
+
+[plugin_config]
+project_id = "my-project"
+dataset = "iggy"
+table = "events"
+mode = "mapped"
+credentials_path = "/secrets/bigquery-writer.json"
+include_metadata = true
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `project_id` | string | required | Google Cloud project that owns the dataset |
+| `dataset` | string | required | BigQuery dataset |
+| `table` | string | required | Target table. It must already exist |
+| `mode` | string | `"mapped"` | `mapped` or `raw` |
+| `payload_column` | string | `"payload"` | Payload column in `raw` mode |
+| `credentials_path` | string | none | Service account key file |
+| `credentials_json` | string | none | Inline service account key. Mutually exclusive with `credentials_path` |
+| `include_metadata` | bool | `true` | Write the `iggy_*` metadata columns listed below |
+| `include_headers` | bool | `false` | Write message headers into `iggy_headers` |
+| `missing_value` | string | `"default"` | How BigQuery fills columns no row in a request sets: `default` or `null` |
+| `max_request_bytes` | usize | `8388608` | Upper bound for one `AppendRows` request, clamped to 64 KiB..9 MiB |
+| `max_retries` | u32 | `3` | Total attempts per call, including the first |
+| `retry_delay` | string | `"1s"` | Base retry delay |
+| `max_retry_delay` | string | `"30s"` | Upper bound for one retry delay |
+| `timeout` | string | `"30s"` | Timeout for `tables.get` and each `AppendRows` call |
+| `verbose_logging` | bool | `false` | Log each batch at info level instead of debug |
+| `endpoint` | string | none | REST endpoint override, for local fakes and emulators. Must be set with `grpc_endpoint`; no credentials are used |
+| `grpc_endpoint` | string | none | Storage Write API `host:port` override |
+
+## Authentication
+
+The connector picks the first source that applies:
+
+1. `endpoint` and `grpc_endpoint`: no credentials, plain-text connections. For tests only.
+2. `credentials_path`: a service account key file.
+3. `credentials_json`: an inline service account key.
+4. Application Default Credentials: `GOOGLE_APPLICATION_CREDENTIALS`, `gcloud auth application-default login`, or the metadata server on GCE, GKE and Cloud Run.
+
+```toml
+# Key file
+credentials_path = "/secrets/bigquery-writer.json"
+
+# Inline key, for example injected through an environment override:
+# IGGY_CONNECTORS_SINK_BIGQUERY_PLUGIN_CONFIG_CREDENTIALS_JSON='{"type":"service_account",...}'
+```
+
+### IAM permissions
+
+The identity needs `bigquery.tables.get` (read the schema) and `bigquery.tables.updateData` (append rows) on the target table. The predefined role `roles/bigquery.dataEditor`, granted on the table or its dataset, includes both.
+
+## Modes
+
+### `mapped` (default)
+
+Each payload must be a JSON object. Top-level fields map to table columns by name. Fields that match no column are ignored.
+
+- **Payload types**: `Json`, `Text` and `Proto` text are accepted when they hold a JSON object. `Raw`, `FlatBuffer` and `Avro` payloads are rejected.
+- **Missing columns**: a column that no row in a request sets is left out of the request, and BigQuery fills it according to `missing_value`. A REQUIRED column without a default value must be present in every row.
+- **Defaults apply per request, not per row**: Arrow has no "unset" marker, so once one row in a request sets a column, the other rows send NULL for it. For a REQUIRED column with a default, rows that omit it while other rows in the same batch set it are rejected. Set such columns in every message or in none.
+- **REPEATED** columns that are absent or `null` are written as empty arrays.
+
+### `raw`
+
+The payload is stored in `payload_column`. Every other table column is left to its default, so a REQUIRED column without a default fails `open()`.
+
+| Payload column type | Accepted payloads |
+| ------------------- | ----------------- |
+| `JSON` | `Json`, plus `Text`, `Proto` and UTF-8 `Raw` payloads that parse as JSON |
+| `STRING` | `Json` (serialized), `Text`, `Proto`, UTF-8 `Raw` |
+| `BYTES` | every payload type, as its raw bytes |
+
+## Metadata Columns
+
+With `include_metadata = true` the table must contain these columns. A missing column fails `open()` with the DDL to add it.
+
+| Column | BigQuery type | Value |
+| ------ | ------------- | ----- |
+| `iggy_stream` | `STRING` | Stream name |
+| `iggy_topic` | `STRING` | Topic name |
+| `iggy_partition_id` | `INT64` | Partition id |
+| `iggy_offset` | `INT64` | Message offset |
+| `iggy_timestamp` | `TIMESTAMP` | Message timestamp |
+| `iggy_id` | `STRING` | Message id, as a decimal string (it is a u128) |
+| `iggy_headers` | `JSON` or `STRING` | Headers, only with `include_headers = true`. Binary values are written as `{"data": "<base64>", "iggy_header_encoding": "base64"}` |
+
+## Type Mapping
+
+| BigQuery | Accepted JSON values |
+| -------- | -------------------- |
+| `STRING`, `GEOGRAPHY` (WKT) | string |
+| `BYTES` | base64 string |
+| `INT64` | integer, or a string holding one |
+| `FLOAT64` | number, or a string holding one |
+| `BOOL` | `true` / `false` |
+| `NUMERIC`, `BIGNUMERIC` | number or string. Prefer strings to avoid float rounding |
+| `TIMESTAMP` | RFC 3339 string, or an integer in microseconds since the epoch |
+| `DATETIME` | `YYYY-MM-DDTHH:MM:SS[.ffffff]` string |
+| `DATE` | `YYYY-MM-DD` string |
+| `TIME` | `HH:MM:SS[.ffffff]` string |
+| `JSON` | any JSON value, stored as that value |
+| `RECORD` | object |
+| `REPEATED` | array |
+
+`INTERVAL` and `RANGE` columns are not supported and fail `open()`.
+
+## Reliability
+
+### Errors and retries
+
+- `UNAVAILABLE`, `CANCELLED`, `DEADLINE_EXCEEDED`, `INTERNAL`, `ABORTED` and `RESOURCE_EXHAUSTED` are retried up to `max_retries` attempts in total.
+- Every other gRPC error, including `INVALID_ARGUMENT`, `PERMISSION_DENIED` and `UNAUTHENTICATED`, fails the request without a retry.
+- `tables.get` retries HTTP 429, 5xx and transport errors. A 403 or 404 fails `open()` immediately.
+
+### Bad rows
+
+A message that cannot become a row (wrong payload type, a value that does not fit its column, a missing REQUIRED value, a row larger than `max_request_bytes`) is dropped before the request is sent. When BigQuery reports row-level errors, it appends nothing from that request: the connector drops the reported rows and appends the remaining rows once more. If that second append also reports row errors, the remaining rows of that request are dropped and the batch returns an error.
+
+Every dropped row is logged at `warn` with its stream, topic, partition and offset. There is no dead-letter queue. On shutdown the connector logs rows written, rejected and failed.
+
+### Delivery semantics
+
+The connector runtime commits consumer group offsets when messages are polled, before `consume()` runs, and a batch for which `consume()` returns an error is not redelivered. As a result:
+
+- Rows in a request that hits a permanent error are lost. Requests split from the same polled batch are attempted independently, so earlier or later chunks may still be written.
+- When the retry budget is exhausted after an ambiguous transport failure, the connector cannot know whether that request was written. Its rows may be absent or duplicated.
+- A process crash between the offset commit and a successful append also loses that batch.
+- The `_default` stream has no append offsets. When an append result is lost and the request is retried, rows can be written twice. Deduplicate downstream on `iggy_stream`, `iggy_topic`, `iggy_partition_id` and `iggy_offset`, for example:
+
+```sql
+SELECT * EXCEPT(rn) FROM (
+  SELECT *, ROW_NUMBER() OVER (
+    PARTITION BY iggy_stream, iggy_topic, iggy_partition_id, iggy_offset

Review Comment:
   warning: offsets restart at 0 after a topic is recreated or purged, so this key collides with old rows and the query hides the new ones. add `iggy_timestamp` to the `PARTITION BY` - a retried duplicate keeps the same timestamp.



##########
core/connectors/sinks/bigquery_sink/src/client.rs:
##########
@@ -0,0 +1,403 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! BigQuery access: credentials, the `tables.get` REST call, and
+//! `AppendRows` on the table's `_default` write stream.
+//!
+//! The schema is fetched over REST with a plain `reqwest` client. Writes go
+//! through `gcloud-bigquery`, which owns the gRPC channel pool and token
+//! refresh for the Storage Write API.
+
+use crate::encode::Chunk;
+use crate::error::{AppendError, TableError};
+use crate::{BigQuerySinkConfig, Settings};
+use gcloud_bigquery::client::google_cloud_auth::credentials::CredentialsFile;
+use gcloud_bigquery::client::google_cloud_auth::token::DefaultTokenSourceProvider;
+use gcloud_bigquery::client::{
+    ChannelConfig, Client, ClientConfig, HttpClientConfig, StreamingWriteConfig,
+};
+use gcloud_bigquery::storage_write::AppendRowsRequestBuilder;
+use gcloud_bigquery::storage_write::stream::default::DefaultStream;
+use gcloud_googleapis::cloud::bigquery::storage::v1::append_rows_response::Response;
+use iggy_connector_sdk::Error;
+use reqwest::header::AUTHORIZATION;
+use secrecy::{ExposeSecret, SecretString};
+use std::fmt;
+use std::sync::Arc;
+use token_source::{TokenSource, TokenSourceProvider};
+use tracing::info;
+
+const BIGQUERY_REST_ENDPOINT: &str = "https://bigquery.googleapis.com";
+
+/// Where credentials come from. Resolved from the config without I/O so the
+/// selection can be unit tested.
+#[derive(Debug, Clone)]
+pub(crate) enum CredentialSource {
+    /// Local fake or emulator: no credentials, plain-text endpoints.
+    Emulator {
+        rest: String,
+        grpc: String,
+    },
+    KeyFile(String),
+    InlineKey(SecretString),
+    ApplicationDefault,
+}
+
+/// Result of one `AppendRows` call that reached BigQuery.
+#[derive(Debug, PartialEq, Eq)]
+pub(crate) enum AppendOutcome {
+    Appended,
+    /// Nothing was appended. Each entry is a row index in the request and
+    /// BigQuery's reason.
+    RowErrors(Vec<(usize, String)>),
+}
+
+pub(crate) struct BigQueryClient {
+    http: reqwest::Client,
+    table_url: String,
+    token: Option<Arc<dyn TokenSource>>,
+    stream: Option<DefaultStream>,
+    write_client: Client,
+    table_path: String,
+    missing_value: i32,
+}
+
+impl CredentialSource {
+    pub(crate) fn from_config(config: &BigQuerySinkConfig) -> Self {
+        if let (Some(rest), Some(grpc)) = (&config.endpoint, &config.grpc_endpoint) {
+            return CredentialSource::Emulator {
+                rest: rest.trim_end_matches('/').to_owned(),
+                grpc: grpc.clone(),
+            };
+        }
+        if let Some(path) = &config.credentials_path {
+            return CredentialSource::KeyFile(path.clone());
+        }
+        if let Some(json) = &config.credentials_json {
+            return CredentialSource::InlineKey(json.clone());
+        }
+        CredentialSource::ApplicationDefault
+    }
+
+    pub(crate) fn label(&self) -> &'static str {
+        match self {
+            CredentialSource::Emulator { .. } => "none (endpoint override)",
+            CredentialSource::KeyFile(_) => "service account key file",
+            CredentialSource::InlineKey(_) => "inline service account key",
+            CredentialSource::ApplicationDefault => "application default credentials",
+        }
+    }
+}
+
+impl BigQueryClient {
+    /// Build credentials and the gRPC channel pool. No BigQuery API call is
+    /// made yet.
+    pub(crate) async fn connect(
+        config: &BigQuerySinkConfig,
+        settings: &Settings,
+    ) -> Result<Self, Error> {
+        install_crypto_provider();
+        let source = CredentialSource::from_config(config);
+        info!("BigQuery sink: using {}", source.label());
+
+        let write_config = StreamingWriteConfig::default()
+            .with_channel_config(ChannelConfig::default().with_timeout(settings.timeout));
+
+        let (client_config, rest_base, token) = match &source {
+            CredentialSource::Emulator { rest, grpc } => (
+                ClientConfig::new_with_emulator(grpc, rest.clone()),
+                rest.clone(),
+                None,
+            ),
+            CredentialSource::KeyFile(path) => {
+                let credentials =
+                    CredentialsFile::new_from_file(path.clone())
+                        .await
+                        .map_err(|e| {
+                            Error::InitError(format!("cannot load credentials_path '{path}': {e}"))
+                        })?;
+                with_credentials(credentials).await?
+            }
+            CredentialSource::InlineKey(json) => {
+                let credentials = CredentialsFile::new_from_str(json.expose_secret())
+                    .await
+                    .map_err(|e| Error::InitError(format!("cannot parse credentials_json: {e}")))?;
+                with_credentials(credentials).await?
+            }
+            CredentialSource::ApplicationDefault => {
+                let (client_config, _) = ClientConfig::new_with_auth().await.map_err(|e| {
+                    Error::InitError(format!("application default credentials: {e}"))
+                })?;
+                let provider = HttpClientConfig::default_token_provider()
+                    .await
+                    .map_err(|e| {
+                        Error::InitError(format!("application default credentials: {e}"))
+                    })?;
+                (
+                    client_config,
+                    BIGQUERY_REST_ENDPOINT.to_owned(),
+                    Some(provider.token_source()),
+                )
+            }
+        };
+
+        let write_client = Client::new(client_config.with_streaming_write_config(write_config))
+            .await
+            .map_err(|e| {
+                Error::InitError(format!("cannot connect to the Storage Write API: {e}"))
+            })?;
+        let http = reqwest::Client::builder()
+            .timeout(settings.timeout)
+            .build()
+            .map_err(|e| Error::InitError(format!("cannot build HTTP client: {e}")))?;
+
+        Ok(BigQueryClient {
+            http,
+            table_url: format!(
+                "{rest_base}/bigquery/v2/projects/{}/datasets/{}/tables/{}?fields=schema",
+                config.project_id, config.dataset, config.table
+            ),
+            token,
+            stream: None,
+            write_client,
+            table_path: format!(
+                "projects/{}/datasets/{}/tables/{}",
+                config.project_id, config.dataset, config.table
+            ),
+            missing_value: settings.missing_value.as_proto(),
+        })
+    }
+
+    /// `tables.get`, returning the raw response body.
+    pub(crate) async fn fetch_table(&self) -> Result<Vec<u8>, TableError> {
+        let mut request = self.http.get(&self.table_url);
+        if let Some(token) = &self.token {
+            let value = token
+                .token()
+                .await
+                .map_err(|e| TableError::Token(e.to_string()))?;
+            request = request.header(AUTHORIZATION, value);
+        }
+        let response = request
+            .send()
+            .await
+            .map_err(|e| TableError::Transport(e.to_string()))?;
+        let status = response.status();
+        let body = response
+            .bytes()
+            .await
+            .map_err(|e| TableError::Transport(e.to_string()))?;
+        if !status.is_success() {
+            return Err(TableError::Http {
+                status: status.as_u16(),
+                body: error_message(&body),
+            });
+        }
+        Ok(body.to_vec())
+    }
+
+    /// Resolve the table's `_default` write stream. Also proves the
+    /// credentials can reach the Storage Write API.
+    pub(crate) async fn resolve_write_stream(&self) -> Result<DefaultStream, AppendError> {
+        Ok(self
+            .write_client
+            .default_storage_writer()
+            .create_write_stream(&self.table_path)
+            .await?)
+    }
+
+    pub(crate) fn set_write_stream(&mut self, stream: DefaultStream) {
+        self.stream = Some(stream);
+    }
+
+    /// One `AppendRows` request for one chunk.
+    pub(crate) async fn append(&self, chunk: &Chunk) -> Result<AppendOutcome, AppendError> {
+        let stream = self.stream.as_ref().ok_or(AppendError::NoResponse)?;
+        let request = AppendRowsRequestBuilder::new_arrow(
+            chunk.schema_bytes.clone(),
+            chunk.batch_bytes.clone(),
+        )
+        .with_default_missing_value_interpretation(self.missing_value);
+        let mut responses = stream.append_rows(vec![request]).await?;
+        let response = responses.message().await?.ok_or(AppendError::NoResponse)?;

Review Comment:
   warning: the channel `timeout` only covers the wait for response headers, so if the stream stalls after them, `responses.message()` waits forever and the hung `consume` blocks `close()` and runtime shutdown. put one deadline around the whole append and set HTTP/2 keepalive.



##########
core/connectors/sinks/bigquery_sink/src/client.rs:
##########
@@ -0,0 +1,403 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! BigQuery access: credentials, the `tables.get` REST call, and
+//! `AppendRows` on the table's `_default` write stream.
+//!
+//! The schema is fetched over REST with a plain `reqwest` client. Writes go
+//! through `gcloud-bigquery`, which owns the gRPC channel pool and token
+//! refresh for the Storage Write API.
+
+use crate::encode::Chunk;
+use crate::error::{AppendError, TableError};
+use crate::{BigQuerySinkConfig, Settings};
+use gcloud_bigquery::client::google_cloud_auth::credentials::CredentialsFile;
+use gcloud_bigquery::client::google_cloud_auth::token::DefaultTokenSourceProvider;
+use gcloud_bigquery::client::{
+    ChannelConfig, Client, ClientConfig, HttpClientConfig, StreamingWriteConfig,
+};
+use gcloud_bigquery::storage_write::AppendRowsRequestBuilder;
+use gcloud_bigquery::storage_write::stream::default::DefaultStream;
+use gcloud_googleapis::cloud::bigquery::storage::v1::append_rows_response::Response;
+use iggy_connector_sdk::Error;
+use reqwest::header::AUTHORIZATION;
+use secrecy::{ExposeSecret, SecretString};
+use std::fmt;
+use std::sync::Arc;
+use token_source::{TokenSource, TokenSourceProvider};
+use tracing::info;
+
+const BIGQUERY_REST_ENDPOINT: &str = "https://bigquery.googleapis.com";
+
+/// Where credentials come from. Resolved from the config without I/O so the
+/// selection can be unit tested.
+#[derive(Debug, Clone)]
+pub(crate) enum CredentialSource {
+    /// Local fake or emulator: no credentials, plain-text endpoints.
+    Emulator {
+        rest: String,
+        grpc: String,
+    },
+    KeyFile(String),
+    InlineKey(SecretString),
+    ApplicationDefault,
+}
+
+/// Result of one `AppendRows` call that reached BigQuery.
+#[derive(Debug, PartialEq, Eq)]
+pub(crate) enum AppendOutcome {
+    Appended,
+    /// Nothing was appended. Each entry is a row index in the request and
+    /// BigQuery's reason.
+    RowErrors(Vec<(usize, String)>),
+}
+
+pub(crate) struct BigQueryClient {
+    http: reqwest::Client,
+    table_url: String,
+    token: Option<Arc<dyn TokenSource>>,
+    stream: Option<DefaultStream>,
+    write_client: Client,
+    table_path: String,
+    missing_value: i32,
+}
+
+impl CredentialSource {
+    pub(crate) fn from_config(config: &BigQuerySinkConfig) -> Self {
+        if let (Some(rest), Some(grpc)) = (&config.endpoint, &config.grpc_endpoint) {

Review Comment:
   warning: with `endpoint` set, configured credentials are ignored, but `validate()` still accepts both - against a real Google endpoint that shows up as an unexplained auth failure. reject the combination in `validate()` and rename `endpoint`/`grpc_endpoint` to `emulator_*` before the first release.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(_) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}

Review Comment:
   warning: arrow-json truncates a fractional JSON number for an INT64 column, so `1.9` lands in BigQuery as `1` with no error (same for DATE and TIME). reject non-integral numbers for these types here, as the README promises.



##########
core/connectors/sinks/bigquery_sink/src/sink.rs:
##########
@@ -0,0 +1,316 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::client::{AppendOutcome, BigQueryClient};
+use crate::encode::{self, Chunk, RunContext};
+use crate::error::{AppendError, TableError};
+use crate::schema::{TableLayout, parse_table_schema};
+use crate::{BigQuerySink, OpenState};
+use async_trait::async_trait;
+use iggy_connector_sdk::retry::{RetryPolicy, retry_async};
+use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata};
+use std::sync::atomic::Ordering;
+use tracing::{debug, error, info, warn};
+
+#[async_trait]
+impl Sink for BigQuerySink {
+    async fn open(&mut self) -> Result<(), Error> {
+        info!(
+            "Opening BigQuery sink connector ID: {} for table {} (mode: {:?})",
+            self.id, self.target, self.settings.mode
+        );
+        // The SDK turns a failed open into a bare status code, so this is the
+        // only place the reason reaches the runtime log.
+        self.try_open().await.inspect_err(|e| {
+            error!(
+                "Failed to open BigQuery sink connector ID: {} for table {}: {e}",
+                self.id, self.target
+            );
+        })
+    }
+
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        if messages.is_empty() {
+            return Ok(());
+        }
+        let Some(state) = &self.state else {
+            return Err(Error::InitError("BigQuery sink is not open".into()));
+        };
+        let received = messages.len();
+        debug!(
+            "BigQuery sink ID: {} received {received} messages from {}/{} partition {} current_offset {}",
+            self.id,
+            topic_metadata.stream,
+            topic_metadata.topic,
+            messages_metadata.partition_id,
+            messages_metadata.current_offset
+        );
+
+        let ctx = RunContext {
+            layout: &state.layout,
+            topic: topic_metadata,
+            messages: &messages_metadata,
+            max_request_bytes: self.settings.max_request_bytes,
+        };
+        let encoded = encode::encode(&ctx, messages).map_err(|e| {
+            self.counters
+                .rows_failed
+                .fetch_add(received as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} cannot encode {received} messages from {}/{} partition {}: {e}",
+                self.id, topic_metadata.stream, topic_metadata.topic, messages_metadata.partition_id
+            );
+            Error::Serialization(e.to_string())
+        })?;
+
+        for rejected in &encoded.rejected {
+            warn!(
+                "BigQuery sink ID: {} dropped message stream={} topic={} partition={} offset={}: {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                rejected.offset,
+                rejected.reason
+            );
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(encoded.rejected.len() as u64, Ordering::Relaxed);
+
+        let mut written = 0u64;
+        let mut last_error = None;
+        for chunk in &encoded.chunks {
+            match self
+                .write_chunk(&state.client, chunk, topic_metadata, &messages_metadata)
+                .await
+            {
+                Ok(rows) => written += rows,
+                Err(e) => last_error = Some(e),
+            }
+        }
+        self.counters
+            .rows_written
+            .fetch_add(written, Ordering::Relaxed);
+
+        if self.settings.verbose {
+            info!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        } else {
+            debug!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        }
+
+        match last_error {

Review Comment:
   warning: when every row of a non-empty batch is rejected, `consume` still returns `Ok`, so the runtime counts the batch as processed even though zero rows landed. return `Error::InvalidRecordValue` with the dropped count, like `http_sink` does.



##########
core/connectors/sinks/bigquery_sink/src/error.rs:
##########
@@ -0,0 +1,209 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Failure classification for BigQuery calls.
+//!
+//! Only transient failures are retried. Retrying a permanent one (bad
+//! schema, missing permission) cannot succeed and only delays the error.
+
+use crate::schema::SchemaError;
+use gcloud_gax::grpc::{Code, Status};
+use iggy_connector_sdk::Error;
+use std::fmt;
+
+/// An `AppendRows` call that did not append. Row-level errors are not part
+/// of this type: they are an expected outcome the caller handles by
+/// dropping the reported rows.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum AppendError {
+    /// The call failed, or the response carried an error status.
+    Rpc { code: Code, message: String },
+    /// The response stream closed without a response.
+    NoResponse,
+}
+
+/// A `tables.get` call that did not return a usable schema.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum TableError {
+    Http { status: u16, body: String },
+    Transport(String),
+    Token(String),
+    Schema(SchemaError),

Review Comment:
   simplification: `TableError::Schema` only carries a `SchemaError` into `Error::InitError`, and `is_retryable` never sees it. map it with `Error::InitError(e.to_string())` in `try_open` and drop the variant, its arms and the `From` impl.



##########
core/connectors/sinks/bigquery_sink/src/lib.rs:
##########
@@ -0,0 +1,461 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Google BigQuery sink connector.
+//!
+//! Writes Iggy messages into a BigQuery table through the Storage Write API
+//! `_default` stream, with rows encoded as Arrow record batches.
+//!
+//! Module map:
+//! - `schema`: BigQuery table schema (from `tables.get`) to Arrow schema.
+//! - `encode`: `ConsumedMessage`s to Arrow record batches, split by size.
+//! - `client`: credentials, `tables.get`, and `AppendRows` calls.
+//! - `error`: gRPC status classification into retryable and permanent.
+//! - `sink`: the `Sink` trait implementation.
+
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{Error, sink_connector};
+use secrecy::SecretString;
+use serde::Deserialize;
+use std::str::FromStr;
+use std::sync::atomic::AtomicU64;
+use std::time::Duration;
+use tracing::warn;
+
+mod client;
+mod encode;
+mod error;
+mod schema;
+mod sink;
+
+sink_connector!(BigQuerySink);
+
+const DEFAULT_PAYLOAD_COLUMN: &str = "payload";
+const DEFAULT_MAX_REQUEST_BYTES: usize = 8 * 1024 * 1024;
+/// Hard ceiling below the 10 MB `AppendRows` limit, leaving room for the
+/// request envelope (stream name, writer schema, protobuf framing).
+const MAX_REQUEST_BYTES_CEILING: usize = 9 * 1024 * 1024;
+/// Smallest accepted request budget. Anything lower makes most real rows
+/// unwritable.
+const MIN_REQUEST_BYTES: usize = 64 * 1024;
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_TIMEOUT: &str = "30s";
+
+/// Plugin configuration, deserialized from `[plugin_config]`.
+///
+/// `Deserialize` only. Nothing re-serializes a plugin config, and leaving
+/// `Serialize` off is what keeps `credentials_json` unserializable.
+#[derive(Debug, Clone, Deserialize)]
+pub struct BigQuerySinkConfig {
+    pub project_id: String,
+    pub dataset: String,
+    pub table: String,
+    /// `mapped` (default) or `raw`.
+    pub mode: Option<WriteMode>,
+    /// Path to a service account key file. Mutually exclusive with
+    /// `credentials_json`. When neither is set, Application Default
+    /// Credentials are used.
+    pub credentials_path: Option<String>,
+    /// Inline service account key JSON.
+    pub credentials_json: Option<SecretString>,
+    /// Column holding the payload in `raw` mode (default `payload`).
+    pub payload_column: Option<String>,
+    /// Write `iggy_stream`, `iggy_topic`, `iggy_partition_id`, `iggy_offset`,
+    /// `iggy_timestamp` and `iggy_id` (default `true`).
+    pub include_metadata: Option<bool>,
+    /// Write message headers into an `iggy_headers` column (default `false`).
+    pub include_headers: Option<bool>,
+    /// How BigQuery fills columns absent from a request: `default` (column
+    /// default value, else NULL) or `null`.
+    pub missing_value: Option<MissingValue>,
+    /// Upper bound for one `AppendRows` request (default 8 MiB).
+    pub max_request_bytes: Option<usize>,
+    /// Total attempts per append, including the first (default 3).
+    pub max_retries: Option<u32>,
+    /// Base retry delay, e.g. `"1s"` (default `1s`).
+    pub retry_delay: Option<String>,
+    /// Upper bound for a single retry delay (default `30s`).
+    pub max_retry_delay: Option<String>,
+    /// Per-request timeout for `tables.get` and `AppendRows` (default `30s`).
+    pub timeout: Option<String>,
+    /// Log every batch at info level instead of debug (default `false`).
+    pub verbose_logging: Option<bool>,
+    /// REST endpoint override, e.g. `http://127.0.0.1:9050`. Must be set
+    /// together with `grpc_endpoint`; when both are set no credentials are
+    /// used. Intended for local fakes and emulators.
+    pub endpoint: Option<String>,
+    /// gRPC `host:port` override for the Storage Write API.
+    pub grpc_endpoint: Option<String>,
+}
+
+/// How message payloads map onto table columns.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum WriteMode {
+    /// Top-level JSON fields map to table columns by name.
+    #[default]
+    Mapped,
+    /// The whole payload goes into one column.
+    Raw,
+}
+
+/// Maps to `AppendRowsRequest.default_missing_value_interpretation`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum MissingValue {
+    #[default]
+    Default,
+    Null,
+}
+
+impl MissingValue {
+    /// Wire value of `AppendRowsRequest.MissingValueInterpretation`.
+    pub(crate) fn as_proto(self) -> i32 {

Review Comment:
   nit: `as_proto` hardcodes `1` and `2` - use `append_rows_request::MissingValueInterpretation::NullValue as i32` and `DefaultValue as i32`, which `gcloud_googleapis` already exports.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(_) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Serialize a row for `arrow-json` while substituting empty strings for BYTES.
+/// The decoder still builds the correct null and list structure, and the binary
+/// leaves are replaced from the original base64 values afterwards.
+struct ArrowObject<'a> {
+    value: &'a OwnedValue,
+    fields: &'a Fields,
+}
+
+impl Serialize for ArrowObject<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let OwnedValue::Object(object) = self.value else {
+            return self.value.serialize(serializer);
+        };
+        let mut map = serializer.serialize_map(Some(self.fields.len()))?;
+        for field in self.fields {
+            if let Some(value) = object.get(field.name().as_str()) {
+                map.serialize_entry(
+                    field.name(),
+                    &ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    },
+                )?;
+            }
+        }
+        map.end()
+    }
+}
+
+struct ArrowValue<'a> {
+    value: &'a OwnedValue,
+    data_type: &'a DataType,
+}
+
+impl Serialize for ArrowValue<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        if matches!(self.value, OwnedValue::Static(StaticNode::Null)) {
+            return self.value.serialize(serializer);
+        }
+        match (self.data_type, self.value) {
+            (DataType::Binary, OwnedValue::String(_)) => serializer.serialize_str(""),
+            (DataType::List(field), OwnedValue::Array(items)) => {
+                let mut sequence = serializer.serialize_seq(Some(items.len()))?;
+                for value in items.iter() {
+                    sequence.serialize_element(&ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    })?;
+                }
+                sequence.end()
+            }
+            (DataType::Struct(fields), OwnedValue::Object(_)) => ArrowObject {
+                value: self.value,
+                fields,
+            }
+            .serialize(serializer),
+            _ => self.value.serialize(serializer),
+        }
+    }
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[&OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    let arrow_rows: Vec<_> = rows
+        .iter()
+        .map(|row| ArrowObject {
+            value: row,
+            fields: schema.fields(),
+        })
+        .collect();
+    decoder.serialize(&arrow_rows)?;
+    let batch = decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))?;
+    replace_binary_columns(batch, rows)
+}
+
+fn replace_binary_columns(
+    batch: RecordBatch,
+    rows: &[&OwnedValue],
+) -> Result<RecordBatch, ArrowError> {
+    let schema = batch.schema();
+    let mut columns = Vec::with_capacity(batch.num_columns());
+    for (field, array) in schema.fields().iter().zip(batch.columns()) {
+        if contains_binary(field.data_type()) {
+            let values = rows
+                .iter()
+                .map(|row| object_value(Some(row), field.name()))
+                .collect::<Result<Vec<_>, _>>()?;
+            columns.push(replace_binary_values(field, array, &values)?);
+        } else {
+            columns.push(array.clone());
+        }
+    }
+    RecordBatch::try_new(schema, columns)
+}
+
+fn replace_binary_values(
+    field: &FieldRef,
+    array: &ArrayRef,
+    values: &[Option<&OwnedValue>],
+) -> Result<ArrayRef, ArrowError> {
+    if array.len() != values.len() {
+        return Err(ArrowError::ComputeError(format!(
+            "column '{}' has {} Arrow values for {} JSON values",
+            field.name(),
+            array.len(),
+            values.len()
+        )));
+    }
+    match field.data_type() {
+        DataType::Binary => binary_array(field.name(), values),
+        DataType::List(item) if contains_binary(item.data_type()) => {
+            let list = array
+                .as_any()
+                .downcast_ref::<ListArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut child_values = Vec::with_capacity(list.values().len());
+            for value in values {
+                match value {
+                    None | Some(OwnedValue::Static(StaticNode::Null)) => {}
+                    Some(OwnedValue::Array(items)) => {
+                        child_values.extend(items.iter().map(Some));
+                    }
+                    Some(_) => {
+                        return Err(ArrowError::JsonError(format!(
+                            "column '{}' expects an array",
+                            field.name()
+                        )));
+                    }
+                }
+            }
+            let child = replace_binary_values(item, list.values(), &child_values)?;
+            Ok(Arc::new(ListArray::try_new(
+                item.clone(),
+                list.offsets().clone(),
+                child,
+                list.nulls().cloned(),
+            )?))
+        }
+        DataType::Struct(fields) if contains_binary(field.data_type()) => {
+            let structure = array
+                .as_any()
+                .downcast_ref::<StructArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut children = Vec::with_capacity(fields.len());
+            for (index, child_field) in fields.iter().enumerate() {
+                if contains_binary(child_field.data_type()) {
+                    let child_values = values
+                        .iter()
+                        .map(|value| object_value(*value, child_field.name()))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    children.push(replace_binary_values(
+                        child_field,
+                        structure.column(index),
+                        &child_values,
+                    )?);
+                } else {
+                    children.push(structure.column(index).clone());
+                }
+            }
+            Ok(Arc::new(StructArray::try_new(
+                fields.clone(),
+                children,
+                structure.nulls().cloned(),
+            )?))
+        }
+        _ => Ok(array.clone()),
+    }
+}
+
+fn binary_array(name: &str, values: &[Option<&OwnedValue>]) -> Result<ArrayRef, ArrowError> {
+    let data_capacity = values
+        .iter()
+        .filter_map(|value| match value {
+            Some(OwnedValue::String(text)) => Some(text.len().saturating_mul(3) / 4),
+            _ => None,
+        })
+        .sum();
+    let mut builder = BinaryBuilder::with_capacity(values.len(), data_capacity);
+    for value in values {
+        match value {
+            None | Some(OwnedValue::Static(StaticNode::Null)) => builder.append_null(),
+            Some(OwnedValue::String(text)) => {
+                let mut decoder = base64::read::DecoderReader::new(text.as_bytes(), &BASE64);
+                io::copy(&mut decoder, &mut builder).map_err(|error| {
+                    ArrowError::JsonError(format!("column '{name}': invalid base64: {error}"))
+                })?;
+                builder.append_value(&[] as &[u8]);
+            }
+            Some(_) => {
+                return Err(ArrowError::JsonError(format!(
+                    "column '{name}' expects a base64 string"
+                )));
+            }
+        }
+    }
+    Ok(Arc::new(builder.finish()))
+}
+
+fn object_value<'a>(
+    value: Option<&'a OwnedValue>,
+    name: &str,
+) -> Result<Option<&'a OwnedValue>, ArrowError> {
+    match value {
+        None | Some(OwnedValue::Static(StaticNode::Null)) => Ok(None),
+        Some(OwnedValue::Object(object)) => Ok(object.get(name)),
+        Some(_) => Err(ArrowError::JsonError(format!(
+            "column '{name}' has a non-object parent"
+        ))),
+    }
+}
+
+fn contains_binary(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Binary => true,
+        DataType::List(field) => contains_binary(field.data_type()),
+        DataType::Struct(fields) => fields
+            .iter()
+            .any(|field| contains_binary(field.data_type())),
+        _ => false,
+    }
+}
+
+fn type_mismatch(field: &FieldRef, array: &ArrayRef) -> ArrowError {
+    ArrowError::ComputeError(format!(
+        "column '{}' has Arrow type {}, expected {}",
+        field.name(),
+        array.data_type(),
+        field.data_type()
+    ))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut builder = BinaryBuilder::with_capacity(messages.len(), 0);
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match payload.try_into_vec() {
+                    Ok(bytes) => {
+                        builder.append_value(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(builder.finish())
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_borrowed_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => {
+                let mut builder = StringBuilder::with_capacity(rows, rows.saturating_mul(39));
+                for message in messages {
+                    write!(&mut builder, "{}", message.id).map_err(|_| {
+                        ArrowError::ComputeError("cannot format message ID".to_owned())
+                    })?;
+                    builder.append_value("");
+                }
+                Arc::new(builder.finish())
+            }
+            MetaColumn::Headers => {
+                let required = !field.is_nullable();
+                Arc::new(StringArray::from_iter(
+                    messages.iter().map(|m| headers_json(m, required)),
+                ))
+            }
+        };
+        fields.push(field.clone());
+        columns.push(column);
+    }
+    Ok(())
+}
+
+/// Headers as a JSON object. Text values are written as strings, binary
+/// values as `{"data": <base64>, "iggy_header_encoding": "base64"}`, the same
+/// shape `http_sink` uses.
+fn headers_json(message: &ConsumedMessage, required: bool) -> Option<String> {
+    let headers = message.headers.as_ref().filter(|h| !h.is_empty());
+    let Some(headers) = headers else {
+        return required.then(|| "{}".to_owned());
+    };
+    let mut object = serde_json::Map::with_capacity(headers.len());
+    for (key, value) in headers {
+        let encoded = match value.as_raw() {
+            Ok(raw) => serde_json::json!({
+                "data": BASE64.encode(raw),
+                "iggy_header_encoding": HEADER_ENCODING_BASE64,
+            }),
+            Err(_) => serde_json::Value::String(value.to_string_value()),
+        };
+        object.insert(key.to_string_value(), encoded);
+    }
+    Some(serde_json::Value::Object(object).to_string())
+}
+
+// ─── Request sizing ──────────────────────────────────────────────────────────
+
+/// Estimate the rows per request from the Arrow allocation, then encode each
+/// candidate chunk. A single row that does not fit is rejected.
+fn split_into_chunks(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<(), ArrowError> {
+    let schema_bytes = encode_schema(batch.schema_ref())?;
+    split_into_chunks_with(
+        batch,
+        offsets,
+        max_bytes,
+        schema_bytes,
+        chunks,
+        rejected,
+        encode_batch,
+    )
+}
+
+fn split_into_chunks_with(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    schema_bytes: Vec<u8>,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+    mut encoder: impl FnMut(&RecordBatch) -> Result<Vec<u8>, ArrowError>,
+) -> Result<(), ArrowError> {
+    let rows_per_chunk = estimated_rows_per_chunk(&batch, schema_bytes.len(), max_bytes);
+    let mut start = 0;
+    while start < batch.num_rows() {
+        let remaining = batch.num_rows() - start;
+        let mut row_count = rows_per_chunk.min(remaining);
+        loop {
+            let candidate = batch.slice(start, row_count);
+            let batch_bytes = encoder(&candidate)?;
+            let size = schema_bytes.len() + batch_bytes.len();
+            if size <= max_bytes {
+                chunks.push(Chunk {
+                    batch: candidate,
+                    offsets: offsets[start..start + row_count].to_vec(),
+                    schema_bytes: schema_bytes.clone(),
+                    batch_bytes,
+                });
+                start += row_count;
+                break;
+            }
+            if row_count == 1 {
+                rejected.push(Rejected {
+                    offset: offsets[start],
+                    reason: format!(
+                        "row is {size} bytes encoded, above max_request_bytes ({max_bytes})"
+                    ),
+                });
+                start += 1;
+                break;
+            }
+
+            let batch_budget = max_bytes.saturating_sub(schema_bytes.len());
+            let smaller = row_count.saturating_mul(batch_budget) / batch_bytes.len().max(1);
+            row_count = smaller.clamp(1, row_count - 1);
+        }
+    }
+    Ok(())
+}
+
+fn estimated_rows_per_chunk(batch: &RecordBatch, schema_bytes: usize, max_bytes: usize) -> usize {
+    if batch.num_rows() == 0 {
+        return 0;
+    }
+    let overhead =
+        IPC_BATCH_OVERHEAD.saturating_add(batch.num_columns().saturating_mul(IPC_COLUMN_OVERHEAD));
+    let batch_budget = max_bytes.saturating_sub(schema_bytes.saturating_add(overhead));
+    let bytes_per_row = batch
+        .get_array_memory_size()

Review Comment:
   warning: this estimate reads buffer capacity, and raw-mode builders over-allocate, so a batch that fits the budget still splits into extra `AppendRows` requests. sum `ArrayData::get_slice_memory_size()` over the columns instead.



##########
core/connectors/sinks/bigquery_sink/README.md:
##########
@@ -0,0 +1,264 @@
+# BigQuery Sink Connector
+
+The BigQuery sink connector consumes messages from Iggy topics and writes them into a Google BigQuery table through the [Storage Write API](https://cloud.google.com/bigquery/docs/write-api) `_default` stream, with rows encoded as Arrow record batches.
+
+## Features
+
+- **Storage Write API**: gRPC `AppendRows` on the table's `_default` stream. No legacy `insertAll`.
+- **Two modes**: `mapped` maps top-level JSON fields to table columns by name. `raw` stores the whole payload in one JSON, STRING or BYTES column.
+- **Schema validation at startup**: the table schema is read with `tables.get` and checked before any data flows. Unsupported column types fail `open()`.
+- **Iggy metadata columns**: stream, topic, partition, offset, timestamp and message id, plus optional headers, for lineage and deduplication.
+- **Per-row error handling**: a row that cannot be written is dropped and logged with its offset. The rest of the batch is still written.
+- **Request sizing**: batches are split below the 10 MB `AppendRows` limit.
+- **Retries**: transient gRPC failures are retried with jittered exponential backoff.
+
+## Configuration
+
+```toml
+type = "sink"
+key = "bigquery"
+enabled = true
+version = 0
+name = "BigQuery sink"
+path = "target/release/libiggy_connector_bigquery_sink"
+
+[[streams]]
+stream = "example_stream"
+topics = ["example_topic"]
+schema = "json"
+batch_length = 1000
+poll_interval = "5ms"
+consumer_group = "bigquery_sink_connector"
+
+[plugin_config]
+project_id = "my-project"
+dataset = "iggy"
+table = "events"
+mode = "mapped"
+credentials_path = "/secrets/bigquery-writer.json"
+include_metadata = true
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+| ------ | ---- | ------- | ----------- |
+| `project_id` | string | required | Google Cloud project that owns the dataset |
+| `dataset` | string | required | BigQuery dataset |
+| `table` | string | required | Target table. It must already exist |
+| `mode` | string | `"mapped"` | `mapped` or `raw` |
+| `payload_column` | string | `"payload"` | Payload column in `raw` mode |
+| `credentials_path` | string | none | Service account key file |
+| `credentials_json` | string | none | Inline service account key. Mutually exclusive with `credentials_path` |
+| `include_metadata` | bool | `true` | Write the `iggy_*` metadata columns listed below |
+| `include_headers` | bool | `false` | Write message headers into `iggy_headers` |
+| `missing_value` | string | `"default"` | How BigQuery fills columns no row in a request sets: `default` or `null` |
+| `max_request_bytes` | usize | `8388608` | Upper bound for one `AppendRows` request, clamped to 64 KiB..9 MiB |
+| `max_retries` | u32 | `3` | Total attempts per call, including the first |
+| `retry_delay` | string | `"1s"` | Base retry delay |
+| `max_retry_delay` | string | `"30s"` | Upper bound for one retry delay |
+| `timeout` | string | `"30s"` | Timeout for `tables.get` and each `AppendRows` call |
+| `verbose_logging` | bool | `false` | Log each batch at info level instead of debug |
+| `endpoint` | string | none | REST endpoint override, for local fakes and emulators. Must be set with `grpc_endpoint`; no credentials are used |
+| `grpc_endpoint` | string | none | Storage Write API `host:port` override |
+
+## Authentication
+
+The connector picks the first source that applies:
+
+1. `endpoint` and `grpc_endpoint`: no credentials, plain-text connections. For tests only.
+2. `credentials_path`: a service account key file.
+3. `credentials_json`: an inline service account key.
+4. Application Default Credentials: `GOOGLE_APPLICATION_CREDENTIALS`, `gcloud auth application-default login`, or the metadata server on GCE, GKE and Cloud Run.
+
+```toml
+# Key file
+credentials_path = "/secrets/bigquery-writer.json"
+
+# Inline key, for example injected through an environment override:
+# IGGY_CONNECTORS_SINK_BIGQUERY_PLUGIN_CONFIG_CREDENTIALS_JSON='{"type":"service_account",...}'
+```
+
+### IAM permissions
+
+The identity needs `bigquery.tables.get` (read the schema) and `bigquery.tables.updateData` (append rows) on the target table. The predefined role `roles/bigquery.dataEditor`, granted on the table or its dataset, includes both.
+
+## Modes
+
+### `mapped` (default)
+
+Each payload must be a JSON object. Top-level fields map to table columns by name. Fields that match no column are ignored.
+
+- **Payload types**: `Json`, `Text` and `Proto` text are accepted when they hold a JSON object. `Raw`, `FlatBuffer` and `Avro` payloads are rejected.
+- **Missing columns**: a column that no row in a request sets is left out of the request, and BigQuery fills it according to `missing_value`. A REQUIRED column without a default value must be present in every row.
+- **Defaults apply per request, not per row**: Arrow has no "unset" marker, so once one row in a request sets a column, the other rows send NULL for it. For a REQUIRED column with a default, rows that omit it while other rows in the same batch set it are rejected. Set such columns in every message or in none.
+- **REPEATED** columns that are absent or `null` are written as empty arrays.
+
+### `raw`
+
+The payload is stored in `payload_column`. Every other table column is left to its default, so a REQUIRED column without a default fails `open()`.
+
+| Payload column type | Accepted payloads |
+| ------------------- | ----------------- |
+| `JSON` | `Json`, plus `Text`, `Proto` and UTF-8 `Raw` payloads that parse as JSON |
+| `STRING` | `Json` (serialized), `Text`, `Proto`, UTF-8 `Raw` |
+| `BYTES` | every payload type, as its raw bytes |
+
+## Metadata Columns
+
+With `include_metadata = true` the table must contain these columns. A missing column fails `open()` with the DDL to add it.
+
+| Column | BigQuery type | Value |
+| ------ | ------------- | ----- |
+| `iggy_stream` | `STRING` | Stream name |
+| `iggy_topic` | `STRING` | Topic name |
+| `iggy_partition_id` | `INT64` | Partition id |
+| `iggy_offset` | `INT64` | Message offset |
+| `iggy_timestamp` | `TIMESTAMP` | Message timestamp |
+| `iggy_id` | `STRING` | Message id, as a decimal string (it is a u128) |
+| `iggy_headers` | `JSON` or `STRING` | Headers, only with `include_headers = true`. Binary values are written as `{"data": "<base64>", "iggy_header_encoding": "base64"}` |
+
+## Type Mapping
+
+| BigQuery | Accepted JSON values |
+| -------- | -------------------- |
+| `STRING`, `GEOGRAPHY` (WKT) | string |
+| `BYTES` | base64 string |
+| `INT64` | integer, or a string holding one |
+| `FLOAT64` | number, or a string holding one |
+| `BOOL` | `true` / `false` |
+| `NUMERIC`, `BIGNUMERIC` | number or string. Prefer strings to avoid float rounding |
+| `TIMESTAMP` | RFC 3339 string, or an integer in microseconds since the epoch |
+| `DATETIME` | `YYYY-MM-DDTHH:MM:SS[.ffffff]` string |
+| `DATE` | `YYYY-MM-DD` string |
+| `TIME` | `HH:MM:SS[.ffffff]` string |
+| `JSON` | any JSON value, stored as that value |
+| `RECORD` | object |
+| `REPEATED` | array |
+
+`INTERVAL` and `RANGE` columns are not supported and fail `open()`.
+
+## Reliability
+
+### Errors and retries
+
+- `UNAVAILABLE`, `CANCELLED`, `DEADLINE_EXCEEDED`, `INTERNAL`, `ABORTED` and `RESOURCE_EXHAUSTED` are retried up to `max_retries` attempts in total.
+- Every other gRPC error, including `INVALID_ARGUMENT`, `PERMISSION_DENIED` and `UNAUTHENTICATED`, fails the request without a retry.
+- `tables.get` retries HTTP 429, 5xx and transport errors. A 403 or 404 fails `open()` immediately.
+
+### Bad rows
+
+A message that cannot become a row (wrong payload type, a value that does not fit its column, a missing REQUIRED value, a row larger than `max_request_bytes`) is dropped before the request is sent. When BigQuery reports row-level errors, it appends nothing from that request: the connector drops the reported rows and appends the remaining rows once more. If that second append also reports row errors, the remaining rows of that request are dropped and the batch returns an error.
+
+Every dropped row is logged at `warn` with its stream, topic, partition and offset. There is no dead-letter queue. On shutdown the connector logs rows written, rejected and failed.
+
+### Delivery semantics
+
+The connector runtime commits consumer group offsets when messages are polled, before `consume()` runs, and a batch for which `consume()` returns an error is not redelivered. As a result:
+
+- Rows in a request that hits a permanent error are lost. Requests split from the same polled batch are attempted independently, so earlier or later chunks may still be written.
+- When the retry budget is exhausted after an ambiguous transport failure, the connector cannot know whether that request was written. Its rows may be absent or duplicated.
+- A process crash between the offset commit and a successful append also loses that batch.
+- The `_default` stream has no append offsets. When an append result is lost and the request is retried, rows can be written twice. Deduplicate downstream on `iggy_stream`, `iggy_topic`, `iggy_partition_id` and `iggy_offset`, for example:
+
+```sql
+SELECT * EXCEPT(rn) FROM (
+  SELECT *, ROW_NUMBER() OVER (
+    PARTITION BY iggy_stream, iggy_topic, iggy_partition_id, iggy_offset
+    ORDER BY iggy_timestamp
+  ) AS rn
+  FROM `my-project.iggy.events`
+) WHERE rn = 1
+```
+
+### Schema changes
+
+The schema is read once in `open()`. Restart the connector after changing the table.
+
+## Testing
+
+Unit tests cover configuration, credential source selection, schema mapping, Arrow encoding, request splitting and error classification.
+
+`tests/bigquery_sink.rs` runs the sink against an in-process fake (`tests/common/mod.rs`): an axum `tables.get` endpoint and a tonic `BigQueryWrite` service that records every `AppendRows` request as an Arrow batch and answers from a script. It covers successful writes, request splitting, local and BigQuery-reported row errors, retryable and permanent failures, and `open()` failures. No Google Cloud access is needed.
+
+The fake cannot check what only BigQuery knows: that it accepts the Arrow encoding, its type coercion, and real credentials. Verify those manually as follows.
+
+### Manual verification against BigQuery
+
+1. Create a dataset and table (requires `gcloud` and `bq`):
+
+    ```bash
+    export PROJECT=my-project
+    gcloud auth application-default login
+    gcloud services enable bigquery.googleapis.com bigquerystorage.googleapis.com --project "$PROJECT"
+    bq --project_id="$PROJECT" mk --dataset iggy
+    bq --project_id="$PROJECT" query --use_legacy_sql=false "
+    CREATE TABLE iggy.events (
+      user_id INT64 NOT NULL,
+      event STRING,
+      amount NUMERIC,
+      created TIMESTAMP DEFAULT CURRENT_TIMESTAMP(),
+      attrs JSON,
+      iggy_stream STRING,
+      iggy_topic STRING,
+      iggy_partition_id INT64,
+      iggy_offset INT64,
+      iggy_timestamp TIMESTAMP,
+      iggy_id STRING
+    )"
+    ```
+
+2. Build the binaries and the plugin, and write a runtime config that loads only this connector:
+
+    ```bash
+    cargo build --bin iggy-server --bin iggy --bin iggy-connectors
+    cargo build -p iggy_connector_bigquery_sink
+    mkdir -p /tmp/bq-sink-test/connectors
+    sed 's#^config_dir = .*#config_dir = "/tmp/bq-sink-test/connectors"#' \
+      core/connectors/runtime/example_config/config.toml > /tmp/bq-sink-test/config.toml
+    cat > /tmp/bq-sink-test/connectors/bigquery.toml <<EOF
+    type = "sink"
+    key = "bigquery"
+    enabled = true
+    version = 0
+    name = "BigQuery sink"
+    path = "$PWD/target/debug/libiggy_connector_bigquery_sink"
+
+    [[streams]]
+    stream = "demo_stream"
+    topics = ["demo_topic"]
+    schema = "json"
+    batch_length = 100
+    poll_interval = "100ms"
+    consumer_group = "bigquery_sink_test"
+
+    [plugin_config]
+    project_id = "$PROJECT"
+    dataset = "iggy"
+    table = "events"
+    verbose_logging = true
+    EOF
+    ```
+
+3. Start the server, create the stream, then start the runtime:
+
+    ```bash
+    ./target/debug/iggy-server                                   # terminal 1

Review Comment:
   nit: this starts `iggy-server` without root credentials, so a fresh server generates a random root password and every `iggy`/`iggy` login below fails. start it with `IGGY_ROOT_USERNAME=iggy IGGY_ROOT_PASSWORD=iggy`, as the connectors README does.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(_) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Serialize a row for `arrow-json` while substituting empty strings for BYTES.
+/// The decoder still builds the correct null and list structure, and the binary
+/// leaves are replaced from the original base64 values afterwards.
+struct ArrowObject<'a> {
+    value: &'a OwnedValue,
+    fields: &'a Fields,
+}
+
+impl Serialize for ArrowObject<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let OwnedValue::Object(object) = self.value else {
+            return self.value.serialize(serializer);
+        };
+        let mut map = serializer.serialize_map(Some(self.fields.len()))?;
+        for field in self.fields {
+            if let Some(value) = object.get(field.name().as_str()) {
+                map.serialize_entry(
+                    field.name(),
+                    &ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    },
+                )?;
+            }
+        }
+        map.end()
+    }
+}
+
+struct ArrowValue<'a> {
+    value: &'a OwnedValue,
+    data_type: &'a DataType,
+}
+
+impl Serialize for ArrowValue<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        if matches!(self.value, OwnedValue::Static(StaticNode::Null)) {
+            return self.value.serialize(serializer);
+        }
+        match (self.data_type, self.value) {
+            (DataType::Binary, OwnedValue::String(_)) => serializer.serialize_str(""),
+            (DataType::List(field), OwnedValue::Array(items)) => {
+                let mut sequence = serializer.serialize_seq(Some(items.len()))?;
+                for value in items.iter() {
+                    sequence.serialize_element(&ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    })?;
+                }
+                sequence.end()
+            }
+            (DataType::Struct(fields), OwnedValue::Object(_)) => ArrowObject {
+                value: self.value,
+                fields,
+            }
+            .serialize(serializer),
+            _ => self.value.serialize(serializer),
+        }
+    }
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[&OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    let arrow_rows: Vec<_> = rows
+        .iter()
+        .map(|row| ArrowObject {
+            value: row,
+            fields: schema.fields(),
+        })
+        .collect();
+    decoder.serialize(&arrow_rows)?;
+    let batch = decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))?;
+    replace_binary_columns(batch, rows)
+}
+
+fn replace_binary_columns(
+    batch: RecordBatch,
+    rows: &[&OwnedValue],
+) -> Result<RecordBatch, ArrowError> {
+    let schema = batch.schema();
+    let mut columns = Vec::with_capacity(batch.num_columns());
+    for (field, array) in schema.fields().iter().zip(batch.columns()) {
+        if contains_binary(field.data_type()) {
+            let values = rows
+                .iter()
+                .map(|row| object_value(Some(row), field.name()))
+                .collect::<Result<Vec<_>, _>>()?;
+            columns.push(replace_binary_values(field, array, &values)?);
+        } else {
+            columns.push(array.clone());
+        }
+    }
+    RecordBatch::try_new(schema, columns)
+}
+
+fn replace_binary_values(
+    field: &FieldRef,
+    array: &ArrayRef,
+    values: &[Option<&OwnedValue>],
+) -> Result<ArrayRef, ArrowError> {
+    if array.len() != values.len() {
+        return Err(ArrowError::ComputeError(format!(
+            "column '{}' has {} Arrow values for {} JSON values",
+            field.name(),
+            array.len(),
+            values.len()
+        )));
+    }
+    match field.data_type() {
+        DataType::Binary => binary_array(field.name(), values),
+        DataType::List(item) if contains_binary(item.data_type()) => {
+            let list = array
+                .as_any()
+                .downcast_ref::<ListArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut child_values = Vec::with_capacity(list.values().len());
+            for value in values {
+                match value {
+                    None | Some(OwnedValue::Static(StaticNode::Null)) => {}
+                    Some(OwnedValue::Array(items)) => {
+                        child_values.extend(items.iter().map(Some));
+                    }
+                    Some(_) => {
+                        return Err(ArrowError::JsonError(format!(
+                            "column '{}' expects an array",
+                            field.name()
+                        )));
+                    }
+                }
+            }
+            let child = replace_binary_values(item, list.values(), &child_values)?;
+            Ok(Arc::new(ListArray::try_new(
+                item.clone(),
+                list.offsets().clone(),
+                child,
+                list.nulls().cloned(),
+            )?))
+        }
+        DataType::Struct(fields) if contains_binary(field.data_type()) => {
+            let structure = array
+                .as_any()
+                .downcast_ref::<StructArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut children = Vec::with_capacity(fields.len());
+            for (index, child_field) in fields.iter().enumerate() {
+                if contains_binary(child_field.data_type()) {
+                    let child_values = values
+                        .iter()
+                        .map(|value| object_value(*value, child_field.name()))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    children.push(replace_binary_values(
+                        child_field,
+                        structure.column(index),
+                        &child_values,
+                    )?);
+                } else {
+                    children.push(structure.column(index).clone());
+                }
+            }
+            Ok(Arc::new(StructArray::try_new(
+                fields.clone(),
+                children,
+                structure.nulls().cloned(),
+            )?))
+        }
+        _ => Ok(array.clone()),
+    }
+}
+
+fn binary_array(name: &str, values: &[Option<&OwnedValue>]) -> Result<ArrayRef, ArrowError> {
+    let data_capacity = values
+        .iter()
+        .filter_map(|value| match value {
+            Some(OwnedValue::String(text)) => Some(text.len().saturating_mul(3) / 4),
+            _ => None,
+        })
+        .sum();
+    let mut builder = BinaryBuilder::with_capacity(values.len(), data_capacity);
+    for value in values {
+        match value {
+            None | Some(OwnedValue::Static(StaticNode::Null)) => builder.append_null(),
+            Some(OwnedValue::String(text)) => {
+                let mut decoder = base64::read::DecoderReader::new(text.as_bytes(), &BASE64);
+                io::copy(&mut decoder, &mut builder).map_err(|error| {
+                    ArrowError::JsonError(format!("column '{name}': invalid base64: {error}"))
+                })?;
+                builder.append_value(&[] as &[u8]);
+            }
+            Some(_) => {
+                return Err(ArrowError::JsonError(format!(
+                    "column '{name}' expects a base64 string"
+                )));
+            }
+        }
+    }
+    Ok(Arc::new(builder.finish()))
+}
+
+fn object_value<'a>(
+    value: Option<&'a OwnedValue>,
+    name: &str,
+) -> Result<Option<&'a OwnedValue>, ArrowError> {
+    match value {
+        None | Some(OwnedValue::Static(StaticNode::Null)) => Ok(None),
+        Some(OwnedValue::Object(object)) => Ok(object.get(name)),
+        Some(_) => Err(ArrowError::JsonError(format!(
+            "column '{name}' has a non-object parent"
+        ))),
+    }
+}
+
+fn contains_binary(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Binary => true,
+        DataType::List(field) => contains_binary(field.data_type()),
+        DataType::Struct(fields) => fields
+            .iter()
+            .any(|field| contains_binary(field.data_type())),
+        _ => false,
+    }
+}
+
+fn type_mismatch(field: &FieldRef, array: &ArrayRef) -> ArrowError {
+    ArrowError::ComputeError(format!(
+        "column '{}' has Arrow type {}, expected {}",
+        field.name(),
+        array.data_type(),
+        field.data_type()
+    ))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut builder = BinaryBuilder::with_capacity(messages.len(), 0);
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match payload.try_into_vec() {
+                    Ok(bytes) => {
+                        builder.append_value(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(builder.finish())
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_borrowed_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => {
+                let mut builder = StringBuilder::with_capacity(rows, rows.saturating_mul(39));
+                for message in messages {
+                    write!(&mut builder, "{}", message.id).map_err(|_| {
+                        ArrowError::ComputeError("cannot format message ID".to_owned())
+                    })?;
+                    builder.append_value("");
+                }
+                Arc::new(builder.finish())
+            }
+            MetaColumn::Headers => {
+                let required = !field.is_nullable();
+                Arc::new(StringArray::from_iter(
+                    messages.iter().map(|m| headers_json(m, required)),
+                ))
+            }
+        };
+        fields.push(field.clone());
+        columns.push(column);
+    }
+    Ok(())
+}
+
+/// Headers as a JSON object. Text values are written as strings, binary
+/// values as `{"data": <base64>, "iggy_header_encoding": "base64"}`, the same
+/// shape `http_sink` uses.
+fn headers_json(message: &ConsumedMessage, required: bool) -> Option<String> {
+    let headers = message.headers.as_ref().filter(|h| !h.is_empty());
+    let Some(headers) = headers else {
+        return required.then(|| "{}".to_owned());
+    };
+    let mut object = serde_json::Map::with_capacity(headers.len());
+    for (key, value) in headers {
+        let encoded = match value.as_raw() {
+            Ok(raw) => serde_json::json!({
+                "data": BASE64.encode(raw),
+                "iggy_header_encoding": HEADER_ENCODING_BASE64,
+            }),
+            Err(_) => serde_json::Value::String(value.to_string_value()),
+        };
+        object.insert(key.to_string_value(), encoded);
+    }
+    Some(serde_json::Value::Object(object).to_string())
+}
+
+// ─── Request sizing ──────────────────────────────────────────────────────────
+
+/// Estimate the rows per request from the Arrow allocation, then encode each
+/// candidate chunk. A single row that does not fit is rejected.
+fn split_into_chunks(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<(), ArrowError> {
+    let schema_bytes = encode_schema(batch.schema_ref())?;
+    split_into_chunks_with(
+        batch,
+        offsets,
+        max_bytes,
+        schema_bytes,
+        chunks,
+        rejected,
+        encode_batch,
+    )
+}
+
+fn split_into_chunks_with(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    schema_bytes: Vec<u8>,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+    mut encoder: impl FnMut(&RecordBatch) -> Result<Vec<u8>, ArrowError>,
+) -> Result<(), ArrowError> {
+    let rows_per_chunk = estimated_rows_per_chunk(&batch, schema_bytes.len(), max_bytes);
+    let mut start = 0;
+    while start < batch.num_rows() {
+        let remaining = batch.num_rows() - start;
+        let mut row_count = rows_per_chunk.min(remaining);
+        loop {
+            let candidate = batch.slice(start, row_count);
+            let batch_bytes = encoder(&candidate)?;
+            let size = schema_bytes.len() + batch_bytes.len();
+            if size <= max_bytes {
+                chunks.push(Chunk {
+                    batch: candidate,
+                    offsets: offsets[start..start + row_count].to_vec(),
+                    schema_bytes: schema_bytes.clone(),
+                    batch_bytes,
+                });
+                start += row_count;
+                break;
+            }
+            if row_count == 1 {
+                rejected.push(Rejected {
+                    offset: offsets[start],
+                    reason: format!(
+                        "row is {size} bytes encoded, above max_request_bytes ({max_bytes})"
+                    ),
+                });
+                start += 1;
+                break;
+            }
+
+            let batch_budget = max_bytes.saturating_sub(schema_bytes.len());
+            let smaller = row_count.saturating_mul(batch_budget) / batch_bytes.len().max(1);
+            row_count = smaller.clamp(1, row_count - 1);
+        }
+    }
+    Ok(())
+}
+
+fn estimated_rows_per_chunk(batch: &RecordBatch, schema_bytes: usize, max_bytes: usize) -> usize {
+    if batch.num_rows() == 0 {
+        return 0;
+    }
+    let overhead =
+        IPC_BATCH_OVERHEAD.saturating_add(batch.num_columns().saturating_mul(IPC_COLUMN_OVERHEAD));
+    let batch_budget = max_bytes.saturating_sub(schema_bytes.saturating_add(overhead));
+    let bytes_per_row = batch
+        .get_array_memory_size()
+        .div_ceil(batch.num_rows())
+        .max(1);
+    (batch_budget / bytes_per_row).max(1).min(batch.num_rows())
+}
+
+/// Serialize the schema and the batch as Arrow IPC stream messages, the
+/// format `AppendRowsRequest.arrow_rows` carries.
+fn encode_ipc(batch: &RecordBatch) -> Result<(Vec<u8>, Vec<u8>), ArrowError> {
+    Ok((encode_schema(batch.schema_ref())?, encode_batch(batch)?))
+}
+
+fn encode_schema(schema: &SchemaRef) -> Result<Vec<u8>, ArrowError> {
+    let options = IpcWriteOptions::default();
+    let generator = IpcDataGenerator::default();
+    let mut tracker = DictionaryTracker::new(true);
+    let schema = generator.schema_to_bytes_with_dictionary_tracker(schema, &mut tracker, &options);
+    let mut schema_bytes = Vec::new();
+    write_message(&mut schema_bytes, schema, &options)?;
+    Ok(schema_bytes)
+}
+
+fn encode_batch(batch: &RecordBatch) -> Result<Vec<u8>, ArrowError> {
+    let options = IpcWriteOptions::default();
+    let generator = IpcDataGenerator::default();
+    let mut tracker = DictionaryTracker::new(true);
+    let mut compression = CompressionContext::default();
+    let _ = generator.schema_to_bytes_with_dictionary_tracker(
+        batch.schema_ref(),
+        &mut tracker,
+        &options,
+    );
+
+    let (dictionaries, encoded) =
+        generator.encode(batch, &mut tracker, &options, &mut compression)?;
+    let mut batch_bytes = Vec::new();
+    for dictionary in dictionaries {
+        write_message(&mut batch_bytes, dictionary, &options)?;
+    }
+    write_message(&mut batch_bytes, encoded, &options)?;
+    Ok(batch_bytes)
+}
+
+// ─── Helpers ─────────────────────────────────────────────────────────────────
+
+fn payload_kind(payload: &Payload) -> &'static str {

Review Comment:
   simplification: `payload_kind` repeats what `Payload::schema()` and the `Display` on `Schema` already give you. format `other.schema()` at lines 216 and 779 and delete it.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(_) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Serialize a row for `arrow-json` while substituting empty strings for BYTES.
+/// The decoder still builds the correct null and list structure, and the binary
+/// leaves are replaced from the original base64 values afterwards.
+struct ArrowObject<'a> {
+    value: &'a OwnedValue,
+    fields: &'a Fields,
+}
+
+impl Serialize for ArrowObject<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let OwnedValue::Object(object) = self.value else {
+            return self.value.serialize(serializer);
+        };
+        let mut map = serializer.serialize_map(Some(self.fields.len()))?;
+        for field in self.fields {
+            if let Some(value) = object.get(field.name().as_str()) {
+                map.serialize_entry(
+                    field.name(),
+                    &ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    },
+                )?;
+            }
+        }
+        map.end()
+    }
+}
+
+struct ArrowValue<'a> {
+    value: &'a OwnedValue,
+    data_type: &'a DataType,
+}
+
+impl Serialize for ArrowValue<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        if matches!(self.value, OwnedValue::Static(StaticNode::Null)) {
+            return self.value.serialize(serializer);
+        }
+        match (self.data_type, self.value) {
+            (DataType::Binary, OwnedValue::String(_)) => serializer.serialize_str(""),
+            (DataType::List(field), OwnedValue::Array(items)) => {
+                let mut sequence = serializer.serialize_seq(Some(items.len()))?;
+                for value in items.iter() {
+                    sequence.serialize_element(&ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    })?;
+                }
+                sequence.end()
+            }
+            (DataType::Struct(fields), OwnedValue::Object(_)) => ArrowObject {
+                value: self.value,
+                fields,
+            }
+            .serialize(serializer),
+            _ => self.value.serialize(serializer),
+        }
+    }
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[&OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    let arrow_rows: Vec<_> = rows
+        .iter()
+        .map(|row| ArrowObject {
+            value: row,
+            fields: schema.fields(),
+        })
+        .collect();
+    decoder.serialize(&arrow_rows)?;
+    let batch = decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))?;
+    replace_binary_columns(batch, rows)
+}
+
+fn replace_binary_columns(
+    batch: RecordBatch,
+    rows: &[&OwnedValue],
+) -> Result<RecordBatch, ArrowError> {
+    let schema = batch.schema();
+    let mut columns = Vec::with_capacity(batch.num_columns());
+    for (field, array) in schema.fields().iter().zip(batch.columns()) {
+        if contains_binary(field.data_type()) {
+            let values = rows
+                .iter()
+                .map(|row| object_value(Some(row), field.name()))
+                .collect::<Result<Vec<_>, _>>()?;
+            columns.push(replace_binary_values(field, array, &values)?);
+        } else {
+            columns.push(array.clone());
+        }
+    }
+    RecordBatch::try_new(schema, columns)
+}
+
+fn replace_binary_values(
+    field: &FieldRef,
+    array: &ArrayRef,
+    values: &[Option<&OwnedValue>],
+) -> Result<ArrayRef, ArrowError> {
+    if array.len() != values.len() {
+        return Err(ArrowError::ComputeError(format!(
+            "column '{}' has {} Arrow values for {} JSON values",
+            field.name(),
+            array.len(),
+            values.len()
+        )));
+    }
+    match field.data_type() {
+        DataType::Binary => binary_array(field.name(), values),
+        DataType::List(item) if contains_binary(item.data_type()) => {
+            let list = array
+                .as_any()
+                .downcast_ref::<ListArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut child_values = Vec::with_capacity(list.values().len());
+            for value in values {
+                match value {
+                    None | Some(OwnedValue::Static(StaticNode::Null)) => {}
+                    Some(OwnedValue::Array(items)) => {
+                        child_values.extend(items.iter().map(Some));
+                    }
+                    Some(_) => {
+                        return Err(ArrowError::JsonError(format!(
+                            "column '{}' expects an array",
+                            field.name()
+                        )));
+                    }
+                }
+            }
+            let child = replace_binary_values(item, list.values(), &child_values)?;
+            Ok(Arc::new(ListArray::try_new(
+                item.clone(),
+                list.offsets().clone(),
+                child,
+                list.nulls().cloned(),
+            )?))
+        }
+        DataType::Struct(fields) if contains_binary(field.data_type()) => {
+            let structure = array
+                .as_any()
+                .downcast_ref::<StructArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut children = Vec::with_capacity(fields.len());
+            for (index, child_field) in fields.iter().enumerate() {
+                if contains_binary(child_field.data_type()) {
+                    let child_values = values
+                        .iter()
+                        .map(|value| object_value(*value, child_field.name()))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    children.push(replace_binary_values(
+                        child_field,
+                        structure.column(index),
+                        &child_values,
+                    )?);
+                } else {
+                    children.push(structure.column(index).clone());
+                }
+            }
+            Ok(Arc::new(StructArray::try_new(
+                fields.clone(),
+                children,
+                structure.nulls().cloned(),
+            )?))
+        }
+        _ => Ok(array.clone()),
+    }
+}
+
+fn binary_array(name: &str, values: &[Option<&OwnedValue>]) -> Result<ArrayRef, ArrowError> {
+    let data_capacity = values
+        .iter()
+        .filter_map(|value| match value {
+            Some(OwnedValue::String(text)) => Some(text.len().saturating_mul(3) / 4),
+            _ => None,
+        })
+        .sum();
+    let mut builder = BinaryBuilder::with_capacity(values.len(), data_capacity);
+    for value in values {
+        match value {
+            None | Some(OwnedValue::Static(StaticNode::Null)) => builder.append_null(),
+            Some(OwnedValue::String(text)) => {
+                let mut decoder = base64::read::DecoderReader::new(text.as_bytes(), &BASE64);
+                io::copy(&mut decoder, &mut builder).map_err(|error| {
+                    ArrowError::JsonError(format!("column '{name}': invalid base64: {error}"))
+                })?;
+                builder.append_value(&[] as &[u8]);
+            }
+            Some(_) => {
+                return Err(ArrowError::JsonError(format!(
+                    "column '{name}' expects a base64 string"
+                )));
+            }
+        }
+    }
+    Ok(Arc::new(builder.finish()))
+}
+
+fn object_value<'a>(
+    value: Option<&'a OwnedValue>,
+    name: &str,
+) -> Result<Option<&'a OwnedValue>, ArrowError> {
+    match value {
+        None | Some(OwnedValue::Static(StaticNode::Null)) => Ok(None),
+        Some(OwnedValue::Object(object)) => Ok(object.get(name)),
+        Some(_) => Err(ArrowError::JsonError(format!(
+            "column '{name}' has a non-object parent"
+        ))),
+    }
+}
+
+fn contains_binary(data_type: &DataType) -> bool {
+    match data_type {
+        DataType::Binary => true,
+        DataType::List(field) => contains_binary(field.data_type()),
+        DataType::Struct(fields) => fields
+            .iter()
+            .any(|field| contains_binary(field.data_type())),
+        _ => false,
+    }
+}
+
+fn type_mismatch(field: &FieldRef, array: &ArrayRef) -> ArrowError {
+    ArrowError::ComputeError(format!(
+        "column '{}' has Arrow type {}, expected {}",
+        field.name(),
+        array.data_type(),
+        field.data_type()
+    ))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut builder = BinaryBuilder::with_capacity(messages.len(), 0);
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match payload.try_into_vec() {
+                    Ok(bytes) => {
+                        builder.append_value(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(builder.finish())
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();
+        simd_json::to_borrowed_value(&mut bytes)
+            .map_err(|e| format!("payload is not valid JSON: {e}"))?;
+    }
+    Ok(text)
+}
+
+// ─── Metadata ────────────────────────────────────────────────────────────────
+
+fn append_metadata(
+    ctx: &RunContext<'_>,
+    messages: &[&ConsumedMessage],
+    fields: &mut Vec<FieldRef>,
+    columns: &mut Vec<ArrayRef>,
+) -> Result<(), ArrowError> {
+    let rows = messages.len();
+    for (meta, field) in &ctx.layout.metadata {
+        let column: ArrayRef = match meta {
+            MetaColumn::Stream => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.stream.as_str(),
+                rows,
+            ))),
+            MetaColumn::Topic => Arc::new(StringArray::from_iter_values(std::iter::repeat_n(
+                ctx.topic.topic.as_str(),
+                rows,
+            ))),
+            MetaColumn::PartitionId => Arc::new(Int64Array::from_iter_values(std::iter::repeat_n(
+                i64::from(ctx.messages.partition_id),
+                rows,
+            ))),
+            MetaColumn::Offset => Arc::new(Int64Array::from_iter_values(
+                messages.iter().map(|m| saturating_i64(m.offset)),
+            )),
+            MetaColumn::Timestamp => Arc::new(
+                TimestampMicrosecondArray::from_iter_values(
+                    messages.iter().map(|m| saturating_i64(m.timestamp)),
+                )
+                .with_timezone(UTC),
+            ),
+            MetaColumn::Id => {
+                let mut builder = StringBuilder::with_capacity(rows, rows.saturating_mul(39));
+                for message in messages {
+                    write!(&mut builder, "{}", message.id).map_err(|_| {
+                        ArrowError::ComputeError("cannot format message ID".to_owned())
+                    })?;
+                    builder.append_value("");
+                }
+                Arc::new(builder.finish())
+            }
+            MetaColumn::Headers => {
+                let required = !field.is_nullable();
+                Arc::new(StringArray::from_iter(
+                    messages.iter().map(|m| headers_json(m, required)),
+                ))
+            }
+        };
+        fields.push(field.clone());
+        columns.push(column);
+    }
+    Ok(())
+}
+
+/// Headers as a JSON object. Text values are written as strings, binary
+/// values as `{"data": <base64>, "iggy_header_encoding": "base64"}`, the same
+/// shape `http_sink` uses.
+fn headers_json(message: &ConsumedMessage, required: bool) -> Option<String> {
+    let headers = message.headers.as_ref().filter(|h| !h.is_empty());
+    let Some(headers) = headers else {
+        return required.then(|| "{}".to_owned());
+    };
+    let mut object = serde_json::Map::with_capacity(headers.len());
+    for (key, value) in headers {
+        let encoded = match value.as_raw() {
+            Ok(raw) => serde_json::json!({
+                "data": BASE64.encode(raw),
+                "iggy_header_encoding": HEADER_ENCODING_BASE64,
+            }),
+            Err(_) => serde_json::Value::String(value.to_string_value()),
+        };
+        object.insert(key.to_string_value(), encoded);
+    }
+    Some(serde_json::Value::Object(object).to_string())
+}
+
+// ─── Request sizing ──────────────────────────────────────────────────────────
+
+/// Estimate the rows per request from the Arrow allocation, then encode each
+/// candidate chunk. A single row that does not fit is rejected.
+fn split_into_chunks(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<(), ArrowError> {
+    let schema_bytes = encode_schema(batch.schema_ref())?;
+    split_into_chunks_with(
+        batch,
+        offsets,
+        max_bytes,
+        schema_bytes,
+        chunks,
+        rejected,
+        encode_batch,
+    )
+}
+
+fn split_into_chunks_with(
+    batch: RecordBatch,
+    offsets: Vec<u64>,
+    max_bytes: usize,
+    schema_bytes: Vec<u8>,
+    chunks: &mut Vec<Chunk>,
+    rejected: &mut Vec<Rejected>,
+    mut encoder: impl FnMut(&RecordBatch) -> Result<Vec<u8>, ArrowError>,
+) -> Result<(), ArrowError> {
+    let rows_per_chunk = estimated_rows_per_chunk(&batch, schema_bytes.len(), max_bytes);
+    let mut start = 0;
+    while start < batch.num_rows() {
+        let remaining = batch.num_rows() - start;
+        let mut row_count = rows_per_chunk.min(remaining);
+        loop {
+            let candidate = batch.slice(start, row_count);
+            let batch_bytes = encoder(&candidate)?;
+            let size = schema_bytes.len() + batch_bytes.len();
+            if size <= max_bytes {
+                chunks.push(Chunk {
+                    batch: candidate,
+                    offsets: offsets[start..start + row_count].to_vec(),
+                    schema_bytes: schema_bytes.clone(),
+                    batch_bytes,
+                });
+                start += row_count;
+                break;
+            }
+            if row_count == 1 {
+                rejected.push(Rejected {
+                    offset: offsets[start],
+                    reason: format!(
+                        "row is {size} bytes encoded, above max_request_bytes ({max_bytes})"
+                    ),
+                });
+                start += 1;
+                break;
+            }
+
+            let batch_budget = max_bytes.saturating_sub(schema_bytes.len());
+            let smaller = row_count.saturating_mul(batch_budget) / batch_bytes.len().max(1);
+            row_count = smaller.clamp(1, row_count - 1);
+        }
+    }
+    Ok(())
+}
+
+fn estimated_rows_per_chunk(batch: &RecordBatch, schema_bytes: usize, max_bytes: usize) -> usize {
+    if batch.num_rows() == 0 {
+        return 0;
+    }
+    let overhead =
+        IPC_BATCH_OVERHEAD.saturating_add(batch.num_columns().saturating_mul(IPC_COLUMN_OVERHEAD));
+    let batch_budget = max_bytes.saturating_sub(schema_bytes.saturating_add(overhead));
+    let bytes_per_row = batch
+        .get_array_memory_size()
+        .div_ceil(batch.num_rows())
+        .max(1);
+    (batch_budget / bytes_per_row).max(1).min(batch.num_rows())
+}
+
+/// Serialize the schema and the batch as Arrow IPC stream messages, the
+/// format `AppendRowsRequest.arrow_rows` carries.
+fn encode_ipc(batch: &RecordBatch) -> Result<(Vec<u8>, Vec<u8>), ArrowError> {
+    Ok((encode_schema(batch.schema_ref())?, encode_batch(batch)?))
+}
+
+fn encode_schema(schema: &SchemaRef) -> Result<Vec<u8>, ArrowError> {
+    let options = IpcWriteOptions::default();
+    let generator = IpcDataGenerator::default();
+    let mut tracker = DictionaryTracker::new(true);
+    let schema = generator.schema_to_bytes_with_dictionary_tracker(schema, &mut tracker, &options);
+    let mut schema_bytes = Vec::new();
+    write_message(&mut schema_bytes, schema, &options)?;
+    Ok(schema_bytes)
+}
+
+fn encode_batch(batch: &RecordBatch) -> Result<Vec<u8>, ArrowError> {
+    let options = IpcWriteOptions::default();
+    let generator = IpcDataGenerator::default();
+    let mut tracker = DictionaryTracker::new(true);
+    let mut compression = CompressionContext::default();
+    let _ = generator.schema_to_bytes_with_dictionary_tracker(

Review Comment:
   simplification: this encodes the schema only to throw it away and prime a dictionary tracker, but no column is ever dictionary-encoded, so the dictionary loop is dead too. call `generator.encode` with a fresh tracker and write just the record batch.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)

Review Comment:
   warning: an explicit `null` in a REQUIRED column with a default passes through here, and arrow-json always rejects the row because the field is non-nullable. drop the `null` key for these columns so BigQuery fills the default.



##########
core/connectors/sinks/bigquery_sink/src/lib.rs:
##########
@@ -0,0 +1,461 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Google BigQuery sink connector.
+//!
+//! Writes Iggy messages into a BigQuery table through the Storage Write API
+//! `_default` stream, with rows encoded as Arrow record batches.
+//!
+//! Module map:
+//! - `schema`: BigQuery table schema (from `tables.get`) to Arrow schema.
+//! - `encode`: `ConsumedMessage`s to Arrow record batches, split by size.
+//! - `client`: credentials, `tables.get`, and `AppendRows` calls.
+//! - `error`: gRPC status classification into retryable and permanent.
+//! - `sink`: the `Sink` trait implementation.
+
+use humantime::Duration as HumanDuration;
+use iggy_connector_sdk::{Error, sink_connector};
+use secrecy::SecretString;
+use serde::Deserialize;
+use std::str::FromStr;
+use std::sync::atomic::AtomicU64;
+use std::time::Duration;
+use tracing::warn;
+
+mod client;
+mod encode;
+mod error;
+mod schema;
+mod sink;
+
+sink_connector!(BigQuerySink);
+
+const DEFAULT_PAYLOAD_COLUMN: &str = "payload";
+const DEFAULT_MAX_REQUEST_BYTES: usize = 8 * 1024 * 1024;
+/// Hard ceiling below the 10 MB `AppendRows` limit, leaving room for the
+/// request envelope (stream name, writer schema, protobuf framing).
+const MAX_REQUEST_BYTES_CEILING: usize = 9 * 1024 * 1024;
+/// Smallest accepted request budget. Anything lower makes most real rows
+/// unwritable.
+const MIN_REQUEST_BYTES: usize = 64 * 1024;
+const DEFAULT_MAX_RETRIES: u32 = 3;
+const DEFAULT_RETRY_DELAY: &str = "1s";
+const DEFAULT_MAX_RETRY_DELAY: &str = "30s";
+const DEFAULT_TIMEOUT: &str = "30s";
+
+/// Plugin configuration, deserialized from `[plugin_config]`.
+///
+/// `Deserialize` only. Nothing re-serializes a plugin config, and leaving
+/// `Serialize` off is what keeps `credentials_json` unserializable.
+#[derive(Debug, Clone, Deserialize)]
+pub struct BigQuerySinkConfig {
+    pub project_id: String,
+    pub dataset: String,
+    pub table: String,
+    /// `mapped` (default) or `raw`.
+    pub mode: Option<WriteMode>,
+    /// Path to a service account key file. Mutually exclusive with
+    /// `credentials_json`. When neither is set, Application Default
+    /// Credentials are used.
+    pub credentials_path: Option<String>,
+    /// Inline service account key JSON.
+    pub credentials_json: Option<SecretString>,
+    /// Column holding the payload in `raw` mode (default `payload`).
+    pub payload_column: Option<String>,
+    /// Write `iggy_stream`, `iggy_topic`, `iggy_partition_id`, `iggy_offset`,
+    /// `iggy_timestamp` and `iggy_id` (default `true`).
+    pub include_metadata: Option<bool>,
+    /// Write message headers into an `iggy_headers` column (default `false`).
+    pub include_headers: Option<bool>,
+    /// How BigQuery fills columns absent from a request: `default` (column
+    /// default value, else NULL) or `null`.
+    pub missing_value: Option<MissingValue>,
+    /// Upper bound for one `AppendRows` request (default 8 MiB).
+    pub max_request_bytes: Option<usize>,
+    /// Total attempts per append, including the first (default 3).
+    pub max_retries: Option<u32>,
+    /// Base retry delay, e.g. `"1s"` (default `1s`).
+    pub retry_delay: Option<String>,
+    /// Upper bound for a single retry delay (default `30s`).
+    pub max_retry_delay: Option<String>,
+    /// Per-request timeout for `tables.get` and `AppendRows` (default `30s`).
+    pub timeout: Option<String>,
+    /// Log every batch at info level instead of debug (default `false`).
+    pub verbose_logging: Option<bool>,
+    /// REST endpoint override, e.g. `http://127.0.0.1:9050`. Must be set
+    /// together with `grpc_endpoint`; when both are set no credentials are
+    /// used. Intended for local fakes and emulators.
+    pub endpoint: Option<String>,
+    /// gRPC `host:port` override for the Storage Write API.
+    pub grpc_endpoint: Option<String>,
+}
+
+/// How message payloads map onto table columns.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum WriteMode {
+    /// Top-level JSON fields map to table columns by name.
+    #[default]
+    Mapped,
+    /// The whole payload goes into one column.
+    Raw,
+}
+
+/// Maps to `AppendRowsRequest.default_missing_value_interpretation`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum MissingValue {
+    #[default]
+    Default,
+    Null,
+}
+
+impl MissingValue {
+    /// Wire value of `AppendRowsRequest.MissingValueInterpretation`.
+    pub(crate) fn as_proto(self) -> i32 {
+        match self {
+            MissingValue::Null => 1,
+            MissingValue::Default => 2,
+        }
+    }
+}
+
+/// Resolved, validated runtime settings. Built once in `new()`.
+#[derive(Debug, Clone)]
+pub(crate) struct Settings {
+    pub mode: WriteMode,
+    pub payload_column: String,
+    pub include_metadata: bool,
+    pub include_headers: bool,
+    pub missing_value: MissingValue,
+    pub max_request_bytes: usize,
+    pub max_retries: u32,
+    pub retry_delay: Duration,
+    pub max_retry_delay: Duration,
+    pub timeout: Duration,
+    pub verbose: bool,
+}
+
+/// Counters updated on the hot path without taking a lock.
+#[derive(Debug, Default)]
+pub(crate) struct Counters {
+    pub rows_written: AtomicU64,
+    pub rows_rejected: AtomicU64,
+    pub rows_failed: AtomicU64,
+}
+
+#[derive(Debug)]
+struct OpenState {
+    client: client::BigQueryClient,
+    layout: schema::TableLayout,
+}
+
+#[derive(Debug)]
+pub struct BigQuerySink {
+    id: u32,
+    config: BigQuerySinkConfig,
+    settings: Settings,
+    /// `project.dataset.table`, safe to log.
+    target: String,
+    state: Option<OpenState>,
+    counters: Counters,
+}
+
+impl BigQuerySink {
+    pub fn new(id: u32, config: BigQuerySinkConfig) -> Self {
+        let settings = Settings::from_config(&config);
+        let target = format!("{}.{}.{}", config.project_id, config.dataset, config.table);
+        BigQuerySink {
+            id,
+            config,
+            settings,
+            target,
+            state: None,
+            counters: Counters::default(),
+        }
+    }
+}
+
+impl Settings {
+    fn from_config(config: &BigQuerySinkConfig) -> Self {
+        let mut retry_delay = parse_duration(config.retry_delay.as_deref(), DEFAULT_RETRY_DELAY);
+        let mut max_retry_delay =
+            parse_duration(config.max_retry_delay.as_deref(), DEFAULT_MAX_RETRY_DELAY);
+        if retry_delay > max_retry_delay {
+            warn!(

Review Comment:
   nit: these lines log without the connector ID, so with two BigQuery sinks running you can't tell which one complained. pass the ID in and include it.
   
   also at lines 210, 275 and `client.rs:115`.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(_) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Serialize a row for `arrow-json` while substituting empty strings for BYTES.
+/// The decoder still builds the correct null and list structure, and the binary
+/// leaves are replaced from the original base64 values afterwards.
+struct ArrowObject<'a> {
+    value: &'a OwnedValue,
+    fields: &'a Fields,
+}
+
+impl Serialize for ArrowObject<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let OwnedValue::Object(object) = self.value else {
+            return self.value.serialize(serializer);
+        };
+        let mut map = serializer.serialize_map(Some(self.fields.len()))?;
+        for field in self.fields {
+            if let Some(value) = object.get(field.name().as_str()) {
+                map.serialize_entry(
+                    field.name(),
+                    &ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    },
+                )?;
+            }
+        }
+        map.end()
+    }
+}
+
+struct ArrowValue<'a> {
+    value: &'a OwnedValue,
+    data_type: &'a DataType,
+}
+
+impl Serialize for ArrowValue<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        if matches!(self.value, OwnedValue::Static(StaticNode::Null)) {
+            return self.value.serialize(serializer);
+        }
+        match (self.data_type, self.value) {
+            (DataType::Binary, OwnedValue::String(_)) => serializer.serialize_str(""),
+            (DataType::List(field), OwnedValue::Array(items)) => {
+                let mut sequence = serializer.serialize_seq(Some(items.len()))?;
+                for value in items.iter() {
+                    sequence.serialize_element(&ArrowValue {
+                        value,
+                        data_type: field.data_type(),
+                    })?;
+                }
+                sequence.end()
+            }
+            (DataType::Struct(fields), OwnedValue::Object(_)) => ArrowObject {
+                value: self.value,
+                fields,
+            }
+            .serialize(serializer),
+            _ => self.value.serialize(serializer),
+        }
+    }
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode(schema: &SchemaRef, rows: &[&OwnedValue]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    let arrow_rows: Vec<_> = rows
+        .iter()
+        .map(|row| ArrowObject {
+            value: row,
+            fields: schema.fields(),
+        })
+        .collect();
+    decoder.serialize(&arrow_rows)?;
+    let batch = decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))?;
+    replace_binary_columns(batch, rows)
+}
+
+fn replace_binary_columns(
+    batch: RecordBatch,
+    rows: &[&OwnedValue],
+) -> Result<RecordBatch, ArrowError> {
+    let schema = batch.schema();
+    let mut columns = Vec::with_capacity(batch.num_columns());
+    for (field, array) in schema.fields().iter().zip(batch.columns()) {
+        if contains_binary(field.data_type()) {
+            let values = rows
+                .iter()
+                .map(|row| object_value(Some(row), field.name()))
+                .collect::<Result<Vec<_>, _>>()?;
+            columns.push(replace_binary_values(field, array, &values)?);
+        } else {
+            columns.push(array.clone());
+        }
+    }
+    RecordBatch::try_new(schema, columns)
+}
+
+fn replace_binary_values(
+    field: &FieldRef,
+    array: &ArrayRef,
+    values: &[Option<&OwnedValue>],
+) -> Result<ArrayRef, ArrowError> {
+    if array.len() != values.len() {
+        return Err(ArrowError::ComputeError(format!(
+            "column '{}' has {} Arrow values for {} JSON values",
+            field.name(),
+            array.len(),
+            values.len()
+        )));
+    }
+    match field.data_type() {
+        DataType::Binary => binary_array(field.name(), values),
+        DataType::List(item) if contains_binary(item.data_type()) => {
+            let list = array
+                .as_any()
+                .downcast_ref::<ListArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut child_values = Vec::with_capacity(list.values().len());
+            for value in values {
+                match value {
+                    None | Some(OwnedValue::Static(StaticNode::Null)) => {}
+                    Some(OwnedValue::Array(items)) => {
+                        child_values.extend(items.iter().map(Some));
+                    }
+                    Some(_) => {
+                        return Err(ArrowError::JsonError(format!(
+                            "column '{}' expects an array",
+                            field.name()
+                        )));
+                    }
+                }
+            }
+            let child = replace_binary_values(item, list.values(), &child_values)?;
+            Ok(Arc::new(ListArray::try_new(
+                item.clone(),
+                list.offsets().clone(),
+                child,
+                list.nulls().cloned(),
+            )?))
+        }
+        DataType::Struct(fields) if contains_binary(field.data_type()) => {
+            let structure = array
+                .as_any()
+                .downcast_ref::<StructArray>()
+                .ok_or_else(|| type_mismatch(field, array))?;
+            let mut children = Vec::with_capacity(fields.len());
+            for (index, child_field) in fields.iter().enumerate() {
+                if contains_binary(child_field.data_type()) {
+                    let child_values = values
+                        .iter()
+                        .map(|value| object_value(*value, child_field.name()))
+                        .collect::<Result<Vec<_>, _>>()?;
+                    children.push(replace_binary_values(
+                        child_field,
+                        structure.column(index),
+                        &child_values,
+                    )?);
+                } else {
+                    children.push(structure.column(index).clone());
+                }
+            }
+            Ok(Arc::new(StructArray::try_new(
+                fields.clone(),
+                children,
+                structure.nulls().cloned(),
+            )?))
+        }
+        _ => Ok(array.clone()),
+    }
+}
+
+fn binary_array(name: &str, values: &[Option<&OwnedValue>]) -> Result<ArrayRef, ArrowError> {
+    let data_capacity = values
+        .iter()
+        .filter_map(|value| match value {
+            Some(OwnedValue::String(text)) => Some(text.len().saturating_mul(3) / 4),
+            _ => None,
+        })
+        .sum();
+    let mut builder = BinaryBuilder::with_capacity(values.len(), data_capacity);
+    for value in values {
+        match value {
+            None | Some(OwnedValue::Static(StaticNode::Null)) => builder.append_null(),
+            Some(OwnedValue::String(text)) => {
+                let mut decoder = base64::read::DecoderReader::new(text.as_bytes(), &BASE64);
+                io::copy(&mut decoder, &mut builder).map_err(|error| {
+                    ArrowError::JsonError(format!("column '{name}': invalid base64: {error}"))
+                })?;
+                builder.append_value(&[] as &[u8]);

Review Comment:
   critical: one `BinaryBuilder` collects a BYTES column for the whole batch, so past `i32::MAX` bytes (about 2 GiB) `append_value` panics, and that panic out of `extern "C"` `iggy_sink_consume` aborts the whole runtime. split the batch by byte budget before building arrays.
   
   also at lines 724, 750.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "kparisa (via GitHub)" <gi...@apache.org>.
kparisa commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4221513462


##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use std::fmt::Write as _;
+use std::io;
+use std::sync::Arc;
+
+use arrow::array::{
+    Array, ArrayRef, BinaryBuilder, Int64Array, ListArray, RecordBatch, StringArray, StringBuilder,
+    StructArray, TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Fields, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use serde::ser::{Serialize, SerializeMap, SerializeSeq, Serializer};
+use simd_json::{OwnedValue, StaticNode};
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values remain base64 until they are decoded directly into
+///   Arrow binary buffers.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)

Review Comment:
   intending to drop such..
   2026-10-08T13:54:06.544169Z  WARN connector: connector_target="iggy_connector_bigquery_sink::sink" BigQuery sink ID: 1 dropped message stream=demo_stream topic=demo_topic partition=0 offset=14: row does not match the table schema: Json error: whilst decoding field 'user_id': failed to parse "bad" as Int64



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "kparisa (via GitHub)" <gi...@apache.org>.
kparisa commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4175782516


##########
core/connectors/sinks/bigquery_sink/tests/bigquery_sink.rs:
##########
@@ -0,0 +1,424 @@
+// Licensed to the Apache Software Foundation (ASF) under one

Review Comment:
   A live BigQuery CI test would require persistent GCP credentials, an enabled billing project, and external resource lifecycle management, and a local emulator is out of scope for this PR.
   I manually verified the full E2E path against a real GCP project: publishing messages to Iggy, running the connector, and confirming the resulting rows in BigQuery. I’ve attached a screenshot from that validation.
   <img width="1392" height="522" alt="Screenshot 2026-10-03 at 3 15 18 PM" src="https://github.com/user-attachments/assets/276bf8b4-7af1-4288-b060-dfbf57ffea99" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(connectors): add Google BigQuery sink connector [iggy]

Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on code in PR #4394:
URL: https://github.com/apache/iggy/pull/4394#discussion_r4178099004


##########
core/connectors/sinks/bigquery_sink/src/error.rs:
##########
@@ -0,0 +1,207 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Failure classification for BigQuery calls.
+//!
+//! Only transient failures are retried. Retrying a permanent one (bad
+//! schema, missing permission) cannot succeed and only delays the error.
+
+use crate::schema::SchemaError;
+use gcloud_gax::grpc::{Code, Status};
+use iggy_connector_sdk::Error;
+use std::fmt;
+
+/// An `AppendRows` call that did not append. Row-level errors are not part
+/// of this type: they are an expected outcome the caller handles by
+/// dropping the reported rows.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum AppendError {
+    /// The call failed, or the response carried an error status.
+    Rpc { code: Code, message: String },
+    /// The response stream closed without a response.
+    NoResponse,
+}
+
+/// A `tables.get` call that did not return a usable schema.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) enum TableError {
+    Http { status: u16, body: String },
+    Transport(String),
+    Token(String),
+    Schema(SchemaError),
+}
+
+impl AppendError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            AppendError::Rpc { code, .. } => is_retryable_code(*code),
+            // The append may or may not have landed. Retrying risks a
+            // duplicate, which the delivery guarantees allow.
+            AppendError::NoResponse => true,
+        }
+    }
+}
+
+impl TableError {
+    pub(crate) fn is_retryable(&self) -> bool {
+        match self {
+            TableError::Http { status, .. } => *status == 429 || *status >= 500,
+            TableError::Transport(_) | TableError::Token(_) => true,
+            TableError::Schema(_) => false,
+        }
+    }
+}
+
+/// gRPC codes BigQuery documents as safe to retry for `AppendRows`.
+pub(crate) fn is_retryable_code(code: Code) -> bool {

Review Comment:
   a client timeout becomes Code::Cancelled, not DeadlineExceeded. is_retryable_code does not retry Cancelled, so append_with_retry returns a permanent error. The runtime already committed the offset (runtime/src/sink.rs:517) and keeps polling. That batch is not replayed. README says DEADLINE_EXCEEDED is retried. **Fix**: retry Code::Cancelled.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, StringBuilder,
+    TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(hex::encode(bytes));

Review Comment:
   mapped BYTES is base64-decoded and then hex-encoded so Arrow can rebuild them. **Fix**: write the decoded bytes into BinaryBuilder



##########
core/connectors/sinks/bigquery_sink/src/sink.rs:
##########
@@ -0,0 +1,303 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::client::{AppendOutcome, BigQueryClient};
+use crate::encode::{self, Chunk, RunContext};
+use crate::error::{AppendError, TableError};
+use crate::schema::{TableLayout, parse_table_schema};
+use crate::{BigQuerySink, OpenState};
+use async_trait::async_trait;
+use iggy_connector_sdk::retry::{RetryPolicy, retry_async};
+use iggy_connector_sdk::{ConsumedMessage, Error, MessagesMetadata, Sink, TopicMetadata};
+use std::sync::atomic::Ordering;
+use tracing::{debug, error, info, warn};
+
+#[async_trait]
+impl Sink for BigQuerySink {
+    async fn open(&mut self) -> Result<(), Error> {
+        info!(
+            "Opening BigQuery sink connector ID: {} for table {} (mode: {:?})",
+            self.id, self.target, self.settings.mode
+        );
+        // The SDK turns a failed open into a bare status code, so this is the
+        // only place the reason reaches the runtime log.
+        self.try_open().await.inspect_err(|e| {
+            error!(
+                "Failed to open BigQuery sink connector ID: {} for table {}: {e}",
+                self.id, self.target
+            );
+        })
+    }
+
+    async fn consume(
+        &self,
+        topic_metadata: &TopicMetadata,
+        messages_metadata: MessagesMetadata,
+        messages: Vec<ConsumedMessage>,
+    ) -> Result<(), Error> {
+        if messages.is_empty() {
+            return Ok(());
+        }
+        let Some(state) = &self.state else {
+            return Err(Error::InitError("BigQuery sink is not open".into()));
+        };
+        let received = messages.len();
+        debug!(
+            "BigQuery sink ID: {} received {received} messages from {}/{} partition {} current_offset {}",
+            self.id,
+            topic_metadata.stream,
+            topic_metadata.topic,
+            messages_metadata.partition_id,
+            messages_metadata.current_offset
+        );
+
+        let ctx = RunContext {
+            layout: &state.layout,
+            topic: topic_metadata,
+            messages: &messages_metadata,
+            max_request_bytes: self.settings.max_request_bytes,
+        };
+        let encoded = encode::encode(&ctx, messages).map_err(|e| {
+            self.counters
+                .rows_failed
+                .fetch_add(received as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} cannot encode {received} messages from {}/{} partition {}: {e}",
+                self.id, topic_metadata.stream, topic_metadata.topic, messages_metadata.partition_id
+            );
+            Error::Serialization(e.to_string())
+        })?;
+
+        for rejected in &encoded.rejected {
+            warn!(
+                "BigQuery sink ID: {} dropped message stream={} topic={} partition={} offset={}: {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                rejected.offset,
+                rejected.reason
+            );
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(encoded.rejected.len() as u64, Ordering::Relaxed);
+
+        let mut written = 0u64;
+        let mut last_error = None;
+        for chunk in &encoded.chunks {
+            match self
+                .write_chunk(&state.client, chunk, topic_metadata, &messages_metadata)
+                .await
+            {
+                Ok(rows) => written += rows,
+                Err(e) => last_error = Some(e),
+            }
+        }
+        self.counters
+            .rows_written
+            .fetch_add(written, Ordering::Relaxed);
+
+        if self.settings.verbose {
+            info!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        } else {
+            debug!(
+                "BigQuery sink ID: {} wrote {written} of {received} messages from {}/{} partition {} to {}",
+                self.id,
+                topic_metadata.stream,
+                topic_metadata.topic,
+                messages_metadata.partition_id,
+                self.target
+            );
+        }
+
+        match last_error {
+            Some(e) => Err(e),
+            None => Ok(()),
+        }
+    }
+
+    async fn close(&mut self) -> Result<(), Error> {
+        self.state = None;
+        info!(
+            "Closed BigQuery sink connector ID: {}, rows written: {}, rows rejected: {}, rows failed: {}",
+            self.id,
+            self.counters.rows_written.load(Ordering::Relaxed),
+            self.counters.rows_rejected.load(Ordering::Relaxed),
+            self.counters.rows_failed.load(Ordering::Relaxed)
+        );
+        Ok(())
+    }
+}
+
+impl BigQuerySink {
+    async fn try_open(&mut self) -> Result<(), Error> {
+        self.config.validate()?;
+
+        let mut client = BigQueryClient::connect(&self.config, &self.settings).await?;
+        let policy = self.retry_policy();
+
+        let context = format!("BigQuery sink ID: {} tables.get", self.id);
+        let body = retry_async(policy, &context, TableError::is_retryable, || {
+            client.fetch_table()
+        })
+        .await
+        .map_err(|failure| Error::InitError(format!("cannot read the table schema: {failure}")))?;
+
+        let columns = parse_table_schema(&body).map_err(|e| Error::from(TableError::Schema(e)))?;
+        let layout = TableLayout::build(columns, &self.settings)
+            .map_err(|e| Error::from(TableError::Schema(e)))?;
+
+        let context = format!("BigQuery sink ID: {} write stream setup", self.id);
+        let stream = retry_async(policy, &context, AppendError::is_retryable, || {
+            client.resolve_write_stream()
+        })
+        .await
+        .map_err(|failure| Error::InitError(format!("cannot open the write stream: {failure}")))?;
+        client.set_write_stream(stream);
+
+        info!(
+            "Opened BigQuery sink connector ID: {} for table {}: {} payload column(s), {} metadata column(s)",
+            self.id,
+            self.target,
+            layout.raw.as_ref().map_or(layout.columns.len(), |_| 1),
+            layout.metadata.len()
+        );
+        self.state = Some(OpenState { client, layout });
+        Ok(())
+    }
+
+    fn retry_policy(&self) -> RetryPolicy {
+        RetryPolicy {
+            max_attempts: self.settings.max_retries,
+            base_delay: self.settings.retry_delay,
+            max_delay: self.settings.max_retry_delay,
+        }
+    }
+
+    /// Append one chunk. When BigQuery reports row errors, nothing was
+    /// appended: drop the reported rows and append the rest once more.
+    /// Returns the number of rows written.
+    async fn write_chunk(
+        &self,
+        client: &BigQueryClient,
+        chunk: &Chunk,
+        topic: &TopicMetadata,
+        messages: &MessagesMetadata,
+    ) -> Result<u64, Error> {
+        let row_errors = match self
+            .append_with_retry(client, chunk, topic, messages)
+            .await?
+        {
+            AppendOutcome::Appended => return Ok(chunk.offsets.len() as u64),
+            AppendOutcome::RowErrors(row_errors) => row_errors,
+        };
+
+        let mut dropped = Vec::with_capacity(row_errors.len());
+        for (row, reason) in row_errors {
+            let offset = chunk.offsets.get(row).copied();
+            warn!(
+                "BigQuery sink ID: {} dropped row rejected by BigQuery stream={} topic={} partition={} offset={}: {reason}",
+                self.id,
+                topic.stream,
+                topic.topic,
+                messages.partition_id,
+                offset.map_or_else(|| format!("unknown (row {row})"), |o| o.to_string())
+            );
+            dropped.push(row);
+        }
+        self.counters
+            .rows_rejected
+            .fetch_add(dropped.len() as u64, Ordering::Relaxed);
+
+        let Some(retry) = encode::without_rows(chunk, &dropped)
+            .map_err(|e| Error::Serialization(e.to_string()))?
+        else {
+            return Ok(0);
+        };
+        match self
+            .append_with_retry(client, &retry, topic, messages)
+            .await?
+        {
+            AppendOutcome::Appended => Ok(retry.offsets.len() as u64),
+            AppendOutcome::RowErrors(again) => {
+                self.counters
+                    .rows_failed
+                    .fetch_add(retry.offsets.len() as u64, Ordering::Relaxed);
+                error!(
+                    "BigQuery sink ID: {} dropped {} rows (offsets {}..={}) of {}/{} partition {}: BigQuery reported {} more row error(s) after the bad rows were removed",
+                    self.id,
+                    retry.offsets.len(),
+                    retry.offsets.first().copied().unwrap_or_default(),
+                    retry.offsets.last().copied().unwrap_or_default(),
+                    topic.stream,
+                    topic.topic,
+                    messages.partition_id,
+                    again.len()
+                );
+                Err(Error::PermanentHttpError(format!(
+                    "{} row error(s) persisted after removing rejected rows",
+                    again.len()
+                )))
+            }
+        }
+    }
+
+    async fn append_with_retry(
+        &self,
+        client: &BigQueryClient,
+        chunk: &Chunk,
+        topic: &TopicMetadata,
+        messages: &MessagesMetadata,
+    ) -> Result<AppendOutcome, Error> {
+        let context = format!(
+            "BigQuery sink ID: {} AppendRows ({} rows)",
+            self.id,
+            chunk.offsets.len()
+        );
+        retry_async(
+            self.retry_policy(),
+            &context,
+            AppendError::is_retryable,
+            || client.append(chunk),
+        )
+        .await
+        .map_err(|failure| {
+            self.counters
+                .rows_failed
+                .fetch_add(chunk.offsets.len() as u64, Ordering::Relaxed);
+            error!(
+                "BigQuery sink ID: {} lost {} rows (offsets {}..={}) of {}/{} partition {}: {failure}",

Review Comment:
   the "lost" log and rows_failed cover every exhausted append error, including NoResponse, which may already be visible. **Fix**: count rows_failed only on permanent failure.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, StringBuilder,
+    TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(hex::encode(bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode<S: serde::Serialize>(schema: &SchemaRef, rows: &[S]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    decoder.serialize(rows)?;
+    decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut values = Vec::with_capacity(messages.len());
+            for message in messages {
+                match message.payload.try_to_bytes() {

Review Comment:
   raw BYTES is cloned, then copied again into the Arrow array. **Fix**: move the bytes into BinaryBuilder.



##########
core/connectors/sinks/bigquery_sink/src/encode.rs:
##########
@@ -0,0 +1,1289 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Turns a run of `ConsumedMessage`s into Arrow record batches that fit in
+//! one `AppendRows` request each.
+//!
+//! A message that cannot become a row (wrong payload type, a value that does
+//! not fit its column, a row larger than the request budget) is rejected on
+//! its own and reported back with its offset. The rest of the batch is still
+//! written.
+//!
+//! Mapped mode decodes JSON objects with `arrow-json`. The fast path decodes
+//! the whole run at once. When that fails, bisection isolates the offending
+//! rows before the survivors are decoded together.
+//!
+//! The writer schema of a request only contains the columns that at least one
+//! row in the run sets. Columns no row mentions are left out, so BigQuery
+//! fills them according to `missing_value` (the column default, or NULL).
+
+use crate::schema::{BqType, Column, MetaColumn, Mode, RawKind, TableLayout, UTC};
+use arrow::array::{
+    ArrayRef, BinaryArray, Int64Array, RecordBatch, StringArray, StringBuilder,
+    TimestampMicrosecondArray,
+};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, FieldRef, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::ipc::writer::{
+    CompressionContext, DictionaryTracker, IpcDataGenerator, IpcWriteOptions, write_message,
+};
+use arrow::json::ReaderBuilder;
+use base64::Engine;
+use base64::engine::general_purpose::STANDARD as BASE64;
+use iggy_connector_sdk::{ConsumedMessage, MessagesMetadata, Payload, TopicMetadata};
+use simd_json::{OwnedValue, StaticNode};
+use std::fmt::Write as _;
+use std::sync::Arc;
+
+const HEADER_ENCODING_BASE64: &str = "base64";
+const UTC_OFFSET: &str = "+00:00";
+const IPC_BATCH_OVERHEAD: usize = 1024;
+const IPC_COLUMN_OVERHEAD: usize = 128;
+
+/// One `AppendRows` request worth of rows.
+#[derive(Debug)]
+pub(crate) struct Chunk {
+    pub batch: RecordBatch,
+    /// Iggy offset of each row, same order as `batch`.
+    pub offsets: Vec<u64>,
+    pub schema_bytes: Vec<u8>,
+    pub batch_bytes: Vec<u8>,
+}
+
+/// A message that was not turned into a row.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(crate) struct Rejected {
+    pub offset: u64,
+    pub reason: String,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct Encoded {
+    pub chunks: Vec<Chunk>,
+    pub rejected: Vec<Rejected>,
+}
+
+/// Per-run context shared by every row.
+pub(crate) struct RunContext<'a> {
+    pub layout: &'a TableLayout,
+    pub topic: &'a TopicMetadata,
+    pub messages: &'a MessagesMetadata,
+    pub max_request_bytes: usize,
+}
+
+/// Encode a run of messages. Fails only on an internal Arrow error, which
+/// means the batch as a whole cannot be built. Per-row problems end up in
+/// `Encoded::rejected`.
+pub(crate) fn encode(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+) -> Result<Encoded, ArrowError> {
+    let mut rejected = Vec::new();
+    let batch = match &ctx.layout.raw {
+        Some((kind, field)) => encode_raw(ctx, *kind, field, messages, &mut rejected)?,
+        None => encode_mapped(ctx, messages, &mut rejected)?,
+    };
+    let mut chunks = Vec::new();
+    if let Some((batch, offsets)) = batch {
+        split_into_chunks(
+            batch,
+            offsets,
+            ctx.max_request_bytes,
+            &mut chunks,
+            &mut rejected,
+        )?;
+    }
+    Ok(Encoded { chunks, rejected })
+}
+
+/// Build a new chunk holding only the rows of `chunk` whose index is not in
+/// `drop`. Used after BigQuery reports row-level errors.
+pub(crate) fn without_rows(chunk: &Chunk, drop: &[usize]) -> Result<Option<Chunk>, ArrowError> {
+    let keep: Vec<bool> = (0..chunk.batch.num_rows())
+        .map(|row| !drop.contains(&row))
+        .collect();
+    let offsets: Vec<u64> = chunk
+        .offsets
+        .iter()
+        .zip(&keep)
+        .filter_map(|(offset, keep)| keep.then_some(*offset))
+        .collect();
+    if offsets.is_empty() {
+        return Ok(None);
+    }
+    let mask = arrow::array::BooleanArray::from(keep);
+    let batch = arrow::compute::filter_record_batch(&chunk.batch, &mask)?;
+    let (schema_bytes, batch_bytes) = encode_ipc(&batch)?;
+    Ok(Some(Chunk {
+        batch,
+        offsets,
+        schema_bytes,
+        batch_bytes,
+    }))
+}
+
+// ─── Mapped mode ─────────────────────────────────────────────────────────────
+
+fn encode_mapped(
+    ctx: &RunContext<'_>,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let layout = ctx.layout;
+    let mut rows = Vec::with_capacity(messages.len());
+    let mut kept = Vec::with_capacity(messages.len());
+    for mut message in messages {
+        let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+        match json_object(payload).and_then(|mut row| {
+            normalize_row(&mut row, layout)?;
+            Ok(row)
+        }) {
+            Ok(row) => {
+                rows.push(row);
+                kept.push(message);
+            }
+            Err(reason) => rejected.push(Rejected {
+                offset: message.offset,
+                reason,
+            }),
+        }
+    }
+    if rows.is_empty() {
+        return Ok(None);
+    }
+
+    let present = present_columns(&rows, layout);
+    if present.is_empty() && layout.metadata.is_empty() {
+        rejected.extend(kept.iter().map(|m| Rejected {
+            offset: m.offset,
+            reason: "payload has no field matching a table column".to_owned(),
+        }));
+        return Ok(None);
+    }
+    let payload_fields: Vec<FieldRef> = present.iter().map(|&i| layout.fields[i].clone()).collect();
+
+    let (decoded, survivors) = decode_rows(&payload_fields, &rows, |row, reason| {
+        rejected.push(Rejected {
+            offset: kept[row].offset,
+            reason,
+        });
+    })?;
+    if survivors.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = survivors.iter().map(|&row| &kept[row]).collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+
+    let mut fields = payload_fields;
+    let mut columns: Vec<ArrayRef> = decoded.map(|b| b.columns().to_vec()).unwrap_or_default();
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Take the JSON object out of a payload. Text and proto text must parse as
+/// a JSON object.
+fn json_object(payload: Payload) -> Result<OwnedValue, String> {
+    let value = match payload {
+        Payload::Json(value) => value,
+        Payload::Text(text) | Payload::Proto(text) => {
+            let mut bytes = text.into_bytes();
+            simd_json::to_owned_value(&mut bytes)
+                .map_err(|e| format!("payload text is not valid JSON: {e}"))?
+        }
+        other => {
+            return Err(format!(
+                "{} payload is not supported in mapped mode",
+                payload_kind(&other)
+            ));
+        }
+    };
+    match value {
+        OwnedValue::Object(_) => Ok(value),
+        _ => Err("payload is not a JSON object".to_owned()),
+    }
+}
+
+/// Prepare one row for `arrow-json`:
+/// - a REQUIRED column without a default must be present and non-null,
+/// - REPEATED columns that are absent or null become `[]`,
+/// - JSON column values become their JSON text,
+/// - BYTES column values are base64 in JSON and become hex, which is what
+///   `arrow-json` decodes into binary.
+fn normalize_row(row: &mut OwnedValue, layout: &TableLayout) -> Result<(), String> {
+    let OwnedValue::Object(object) = row else {
+        return Err("payload is not a JSON object".to_owned());
+    };
+    for column in &layout.columns {
+        let value = object.get_mut(column.name.as_str());
+        match (value, column.mode) {
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                object.insert(column.name.clone(), empty_array());
+            }
+            (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Required)
+                if !column.has_default =>
+            {
+                return Err(format!(
+                    "missing value for REQUIRED column '{}'",
+                    column.name
+                ));
+            }
+            (Some(value), _) => normalize_value(value, column)?,
+            (None, _) => {}
+        }
+    }
+    Ok(())
+}
+
+fn normalize_value(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if column.mode == Mode::Repeated {
+        let OwnedValue::Array(items) = value else {
+            return Err(format!("column '{}' expects an array", column.name));
+        };
+        for item in items.iter_mut() {
+            normalize_scalar(item, column)?;
+        }
+        return Ok(());
+    }
+    normalize_scalar(value, column)
+}
+
+fn normalize_scalar(value: &mut OwnedValue, column: &Column) -> Result<(), String> {
+    if matches!(value, OwnedValue::Static(StaticNode::Null)) {
+        return Ok(());
+    }
+    match &column.bq_type {
+        BqType::Json => {
+            let text = simd_json::to_string(value)
+                .map_err(|e| format!("column '{}': cannot serialize JSON: {e}", column.name))?;
+            *value = OwnedValue::String(text);
+        }
+        BqType::Bytes => {
+            let OwnedValue::String(text) = value else {
+                return Err(format!("column '{}' expects a base64 string", column.name));
+            };
+            let bytes = BASE64
+                .decode(text.as_bytes())
+                .map_err(|e| format!("column '{}': invalid base64: {e}", column.name))?;
+            *value = OwnedValue::String(hex::encode(bytes));
+        }
+        BqType::Record(children) => {
+            let OwnedValue::Object(object) = value else {
+                return Ok(());
+            };
+            for child in children {
+                match (object.get_mut(child.name.as_str()), child.mode) {
+                    (None | Some(OwnedValue::Static(StaticNode::Null)), Mode::Repeated) => {
+                        object.insert(child.name.clone(), empty_array());
+                    }
+                    (Some(child_value), _) => normalize_value(child_value, child)?,
+                    (None, _) => {}
+                }
+            }
+        }
+        _ => {}
+    }
+    Ok(())
+}
+
+/// Indexes of the layout columns that at least one row sets, in table order.
+fn present_columns(rows: &[OwnedValue], layout: &TableLayout) -> Vec<usize> {
+    let mut present = vec![false; layout.columns.len()];
+    for row in rows {
+        if let OwnedValue::Object(object) = row {
+            for key in object.keys() {
+                if let Some(&index) = layout.index.get(key.as_str()) {
+                    present[index] = true;
+                }
+            }
+        }
+    }
+    present
+        .iter()
+        .enumerate()
+        .filter_map(|(index, set)| set.then_some(index))
+        .collect()
+}
+
+/// Decode rows into one batch. Returns the batch (`None` when there are no
+/// payload fields) and the indexes of the rows it holds.
+fn decode_rows(
+    fields: &[FieldRef],
+    rows: &[OwnedValue],
+    on_reject: impl FnMut(usize, String),
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    decode_rows_with(fields, rows, on_reject, decode)
+}
+
+fn decode_rows_with<'a>(
+    fields: &[FieldRef],
+    rows: &'a [OwnedValue],
+    mut on_reject: impl FnMut(usize, String),
+    mut decoder: impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(Option<RecordBatch>, Vec<usize>), ArrowError> {
+    if fields.is_empty() || rows.is_empty() {
+        return Ok((None, (0..rows.len()).collect()));
+    }
+    let target: SchemaRef = Arc::new(Schema::new(fields.to_vec()));
+    let schema: SchemaRef = Arc::new(Schema::new(
+        fields
+            .iter()
+            .map(|f| {
+                f.as_ref()
+                    .clone()
+                    .with_data_type(decode_type(f.data_type()))
+            })
+            .collect::<Vec<_>>(),
+    ));
+    let row_refs: Vec<&OwnedValue> = rows.iter().collect();
+    let first_error = match decoder(&schema, &row_refs) {
+        Ok(batch) => {
+            return Ok((Some(to_target(batch, &target)?), (0..rows.len()).collect()));
+        }
+        Err(error) => error,
+    };
+
+    let mut valid = vec![true; rows.len()];
+    isolate_invalid_rows(
+        &schema,
+        &row_refs,
+        0,
+        first_error,
+        &mut valid,
+        &mut on_reject,
+        &mut decoder,
+    )?;
+
+    let mut valid_rows = Vec::with_capacity(rows.len());
+    let mut survivors = Vec::with_capacity(rows.len());
+    for (index, row) in row_refs.into_iter().enumerate() {
+        if valid[index] {
+            valid_rows.push(row);
+            survivors.push(index);
+        }
+    }
+    if valid_rows.is_empty() {
+        return Ok((None, survivors));
+    }
+    let batch = decoder(&schema, &valid_rows)?;
+    Ok((Some(to_target(batch, &target)?), survivors))
+}
+
+fn isolate_invalid_rows<'a>(
+    schema: &SchemaRef,
+    rows: &[&'a OwnedValue],
+    first_index: usize,
+    error: ArrowError,
+    valid: &mut [bool],
+    on_reject: &mut impl FnMut(usize, String),
+    decoder: &mut impl FnMut(&SchemaRef, &[&'a OwnedValue]) -> Result<RecordBatch, ArrowError>,
+) -> Result<(), ArrowError> {
+    if rows.len() == 1 {
+        valid[first_index] = false;
+        on_reject(
+            first_index,
+            format!("row does not match the table schema: {error}"),
+        );
+        return Ok(());
+    }
+
+    let midpoint = rows.len() / 2;
+    let (left, right) = rows.split_at(midpoint);
+    if let Err(error) = decoder(schema, left) {
+        isolate_invalid_rows(schema, left, first_index, error, valid, on_reject, decoder)?;
+    }
+    if let Err(error) = decoder(schema, right) {
+        isolate_invalid_rows(
+            schema,
+            right,
+            first_index + midpoint,
+            error,
+            valid,
+            on_reject,
+            decoder,
+        )?;
+    }
+    Ok(())
+}
+
+/// The type `arrow-json` decodes into. Without the `chrono-tz` feature it
+/// only parses offset timezones, so `"UTC"` is decoded as `"+00:00"` and
+/// relabelled afterwards. Both name the same instant, so the relabel is a
+/// metadata change.
+fn decode_type(data_type: &DataType) -> DataType {
+    match data_type {
+        DataType::Timestamp(unit, Some(_)) => DataType::Timestamp(*unit, Some(UTC_OFFSET.into())),
+        DataType::List(item) => DataType::List(Arc::new(
+            item.as_ref()
+                .clone()
+                .with_data_type(decode_type(item.data_type())),
+        )),
+        DataType::Struct(children) => DataType::Struct(
+            children
+                .iter()
+                .map(|f| {
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(decode_type(f.data_type()))
+                })
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+fn to_target(batch: RecordBatch, target: &SchemaRef) -> Result<RecordBatch, ArrowError> {
+    let columns = batch
+        .columns()
+        .iter()
+        .zip(target.fields())
+        .map(|(column, field)| {
+            if column.data_type() == field.data_type() {
+                Ok(column.clone())
+            } else {
+                cast(column, field.data_type())
+            }
+        })
+        .collect::<Result<Vec<_>, _>>()?;
+    RecordBatch::try_new(target.clone(), columns)
+}
+
+fn decode<S: serde::Serialize>(schema: &SchemaRef, rows: &[S]) -> Result<RecordBatch, ArrowError> {
+    let mut decoder = ReaderBuilder::new(schema.clone())
+        .with_batch_size(rows.len().max(1))
+        .build_decoder()?;
+    decoder.serialize(rows)?;
+    decoder
+        .flush()?
+        .ok_or_else(|| ArrowError::JsonError("decoder produced no rows".to_owned()))
+}
+
+// ─── Raw mode ────────────────────────────────────────────────────────────────
+
+fn encode_raw(
+    ctx: &RunContext<'_>,
+    kind: RawKind,
+    payload_field: &FieldRef,
+    messages: Vec<ConsumedMessage>,
+    rejected: &mut Vec<Rejected>,
+) -> Result<Option<(RecordBatch, Vec<u64>)>, ArrowError> {
+    let mut kept = Vec::with_capacity(messages.len());
+    let payload_column: ArrayRef = match kind {
+        RawKind::Bytes => {
+            let mut values = Vec::with_capacity(messages.len());
+            for message in messages {
+                match message.payload.try_to_bytes() {
+                    Ok(bytes) => {
+                        values.push(bytes);
+                        kept.push(message);
+                    }
+                    Err(e) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason: e.to_string(),
+                    }),
+                }
+            }
+            Arc::new(BinaryArray::from_iter_values(values))
+        }
+        RawKind::Json | RawKind::String => {
+            let mut values = Vec::with_capacity(messages.len());
+            for mut message in messages {
+                let payload = std::mem::replace(&mut message.payload, Payload::Raw(Vec::new()));
+                match raw_text(payload, kind) {
+                    Ok(text) => {
+                        values.push(text);
+                        kept.push(message);
+                    }
+                    Err(reason) => rejected.push(Rejected {
+                        offset: message.offset,
+                        reason,
+                    }),
+                }
+            }
+            Arc::new(StringArray::from_iter_values(values))
+        }
+    };
+    if kept.is_empty() {
+        return Ok(None);
+    }
+    let kept: Vec<&ConsumedMessage> = kept.iter().collect();
+    let offsets = kept.iter().map(|m| m.offset).collect();
+    let mut fields = vec![payload_field.clone()];
+    let mut columns = vec![payload_column];
+    append_metadata(ctx, &kept, &mut fields, &mut columns)?;
+    let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns)?;
+    Ok(Some((batch, offsets)))
+}
+
+/// Payload as text for a JSON or STRING column. A JSON column only takes
+/// text that parses as JSON.
+fn raw_text(payload: Payload, kind: RawKind) -> Result<String, String> {
+    let text = match payload {
+        Payload::Json(value) => {
+            return simd_json::to_string(&value).map_err(|e| format!("cannot serialize JSON: {e}"));
+        }
+        Payload::Text(text) | Payload::Proto(text) => text,
+        Payload::Raw(bytes) => {
+            String::from_utf8(bytes).map_err(|_| "raw payload is not valid UTF-8".to_owned())?
+        }
+        other => {
+            return Err(format!(
+                "{} payload cannot be written to a text column",
+                payload_kind(&other)
+            ));
+        }
+    };
+    if kind == RawKind::Json {
+        let mut bytes = text.as_bytes().to_vec();

Review Comment:
   raw JSON validation copies the payload into an owned tree and then drops it. **Fix**: use simd_json::to_borrowed_value



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org