You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fizzabid (via GitHub)" <gi...@apache.org> on 2026/08/31 08:38:33 UTC
[PR] Kafka Connect: add upsert, source-timezone shift, and naive-storage options [iceberg]
Fizzabid opened a new pull request, #17899:
URL: https://github.com/apache/iceberg/pull/17899
Adds experimental modes to the kafka-connect sink for CDC pipelines fed by Debezium. All controlled by connector config; defaults preserve upstream behavior.
Upsert (equality-delete backed) — enabled by
`iceberg.tables.upsert-mode-enabled=true` plus per-table `iceberg.tables.<name>.id-columns` (or global `default-id-columns`). New `RecordDeltaWriter` extends Iceberg core's `BaseTaskWriter` with per-partition `BaseEqualityDeltaWriter` instances. `IcebergWriter.write` reads `iceberg.tables.cdc-field` (default `_cdc.op`) and routes I/C/R to insertRow, U to updateRow, D to deleteRow. `RecordUtils.createTableWriter` selects the delta writer when both the flag and id-columns are set.
Source-timezone shift — `iceberg.tables.source-timezone=<zone>` treats Debezium's naive DATETIME/DATETIME2 columns (mislabeled as UTC by the JDBC driver) as the given IANA zone and rewrites incoming epoch millis to the corrected UTC instant via
recover-digits → relabel-as-source-zone → back-to-instant. `RecordConverter.correctForSourceTimezone` is invoked from both the timestamptz and naive Number/Date branches.
Naive storage — `iceberg.tables.store-naive-timestamps=true` forces `SchemaUtils.toIcebergType` to emit `TimestampType.withoutZone()` for Debezium Connect.Timestamp columns. Combined with `source-timezone`, the new `RecordConverter.millisToNaive` stores the source-zone wall-clock digits as a naive `timestamp` (matches SQL Server display; ClickHouse IcebergS3 auto-discovers `DateTime64(6)` without a zone attribute).
Files touched:
- IcebergSinkConfig: new configs and getters
- SchemaUtils: naive vs zoned type inference
- RecordConverter: shift and naive value conversion
- IcebergWriter, RecordUtils: upsert delta-writer routing
- RecordDeltaWriter: new equality-delete writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Kafka Connect: add upsert, source-timezone shift, and naive-storage options [iceberg]
Posted by "Fizzabid (via GitHub)" <gi...@apache.org>.
Fizzabid closed pull request #17899: Kafka Connect: add upsert, source-timezone shift, and naive-storage options
URL: https://github.com/apache/iceberg/pull/17899
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org