You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2026/06/05 20:14:18 UTC
[PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
ahmedabu98 opened a new pull request, #38837:
URL: https://github.com/apache/beam/pull/38837
Adds read transforms that consume planned Iceberg changelog tasks and turn them into Beam CDC rows.
`ReadFromChangelogs` reads task batches produced by `ChangelogScanner`:
- Unidirectional task batches are read and emitted directly as `INSERT` or `DELETE` rows
- Large bidirectional task batches are read with full rows, filtered by primary-key overlap range, and routed into keyed insert/delete streams for downstream update resolution.
- Records outside the overlap range are safely emitted directly because they cannot match an opposing change.
`LocalResolveDoFn` reads and resolves small bidirectional task batches in memory.
`CdcResolver` centralizes the logic for reconciling deletes and inserts for the same primary key. It emits changed pairs as `UPDATE_BEFORE` / `UPDATE_AFTER`, and leaves unmatched rows as `DELETE` or `INSERT`. Duplicate no-op pairs with identical non-PK fields are dropped.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "talatuyarer (via GitHub)" <gi...@apache.org>.
talatuyarer commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3574733689
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
+
+ @Nullable StructLike overlapLower = ovl.toStructLike(descriptor.getOverlapLower());
+ @Nullable StructLike overlapUpper = ovl.toStructLike(descriptor.getOverlapUpper());
+ for (SerializableChangelogTask task : element.getValue()) {
+ readAndRoute(descriptor, task, table, overlapLower, overlapUpper, pkGroups, out);
+ }
+
+ resolveAndEmit(
+ descriptor,
+ pkGroups,
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), table.schema()),
+ out);
+ }
+
+ /**
+ * Processes a {@link SerializableChangelogTask} and routes each record.
+ *
+ * <ul>
+ * <li>Out of overlap: emit directly
+ * <li>Inside overlap: stash in {@code pkGroups} to resolve in {@link #resolveAndEmit}
+ * </ul>
+ */
+ private void readAndRoute(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ Table table,
+ @Nullable StructLike overlapLower,
+ @Nullable StructLike overlapUpper,
+ StructLikeMap<PkGroup> pkGroups,
+ OutputReceiver<Row> out)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, false)) {
+ for (Record rec : records) {
+ if (ovl.contains(rec, overlapLower, overlapUpper)) { // needs resolution
+ StructLike pk = StructLikeUtil.copy(ovl.recordIdProjection());
+ PkGroup group = pkGroups.computeIfAbsent(pk, k -> new PkGroup());
+ if (isInsert) {
+ group.inserts.add(rec);
+ } else {
+ group.deletes.add(rec);
+ }
+ } else { // safe to emit directly
+ emit(descriptor, rec, isInsert ? ValueKind.INSERT : ValueKind.DELETE, out);
+ }
+ }
+ }
+ }
+
+ /** Resolves each PK group using {@link CdcResolver}. */
+ private void resolveAndEmit(
+ ChangelogDescriptor descriptor,
+ StructLikeMap<PkGroup> pkGroups,
+ Schema fullSchema,
+ OutputReceiver<Row> out) {
+ CdcResolver<Record> resolver = new RecordResolver(checkStateNotNull(nonPkFields), fullSchema);
+ for (PkGroup group : pkGroups.values()) {
+ resolver.resolve(
+ group.deletes,
+ group.inserts,
+ (kind, rec) -> {
+ emit(descriptor, rec, kind, out);
+ });
+ }
+ }
+
+ /** Resolver specialization that hashes Iceberg Record non-PK fields. */
+ private static final class RecordResolver extends CdcResolver<Record> {
+ private final List<Types.NestedField> nonPkFields;
+ private final Comparator<StructLike> nonPkComparator;
+ private final StructProjection left;
+ private final StructProjection right;
+
+ RecordResolver(List<Types.NestedField> nonPkFields, Schema recSchema) {
+ this.nonPkFields = nonPkFields;
+ Set<Integer> nonPkFieldIds =
+ nonPkFields.stream().map(Types.NestedField::fieldId).collect(Collectors.toSet());
+ this.left = StructProjection.create(recSchema, nonPkFieldIds);
+ this.right = StructProjection.create(recSchema, nonPkFieldIds);
+ this.nonPkComparator =
+ Comparators.forType(TypeUtil.select(recSchema, nonPkFieldIds).asStruct());
+ }
+
+ @Override
+ protected int nonPkHash(Record rec) {
+ int hash = 1;
+ for (Types.NestedField field : nonPkFields) {
+ hash = 31 * hash + Objects.hashCode(rec.getField(field.name()));
Review Comment:
On a table with a non-PK fixed column, a copy-on-write rewrite's delete/insert pair never matches in the hash index, so unchanged rows are emitted as spurious UPDATE_BEFORE/UPDATE_AFTER pairs.
Can we derive both hash and equals from one mechanism
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
+ outputReceiver
+ .get(taggedOutput)
+ .builder(
+ KV.of(
+ CdcRowDescriptor.builder()
+ .setCommitSnapshotId(commitSnapshotId)
+ .setSnapshotSequenceNumber(commitSnapshotSequenceNumber)
+ .setPrimaryKey(pk)
+ .build(),
+ row))
+ .setValueKind(kind)
+ .output();
+
+ } else {
+ // outside overlap -- get projected record and output
+ StructLike projected = outputProjector().wrap(rec);
+ Row row = structToRow(projectedBeamRowSchema, projected);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ }
+ }
+ }
+
+ trackMetrics(task.getType());
+ }
+
+ private StructProjection outputProjector() {
+ if (outputProjector == null) {
+ outputProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+ return outputProjector;
+ }
+
+ private StructProjection pkProjector() {
+ if (pkProjector == null) {
+ pkProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ scanConfig.recordIdSchema());
+ }
+ return pkProjector;
+ }
+
+ private void trackMetrics(SerializableChangelogTask.Type type) {
+ switch (type) {
+ case ADDED_ROWS:
+ numAddedRowsScanTasksCompleted.inc();
+ break;
+ case DELETED_ROWS:
+ numDeletedRowsScanTasksCompleted.inc();
+ break;
+ case DELETED_FILE:
+ numDeletedDataFileScanTasksCompleted.inc();
+ break;
+ }
+ }
+
+ private String getKind(SerializableChangelogTask.Type taskType) {
+ switch (taskType) {
+ case ADDED_ROWS:
+ return "INSERT";
+ case DELETED_ROWS:
+ return "DELETE";
+ case DELETED_FILE:
+ default:
+ return "DELETE-DF";
+ }
+ }
+
+ private static final int COMPRESSED_TO_DECODED_BYTES_ESTIMATE = 4;
+
+ @GetSize
+ public double getSize(
Review Comment:
The current implementation in `getSize` incorrectly calculates the size by aggregating the full data file size plus all delete files multiplied by four, while ignoring the specific `getStart()` and `getLength()` values.
Since Iceberg splits files into byte-range tasks and the scanner batches by these lengths, this approach results in an inflated size estimate of approximately N × 4 × actual_bytes for a file split into N tasks, which negatively impacts progress reporting and autoscaling.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
Review Comment:
Shuffle key is built via structToRow(rowIdBeamSchema, ...), and timestamptz maps to Beam DATETIME (joda millis). Two distinct PKs differing only in sub-millisecond digits collide into one CdcRowDescriptor group and get paired as an update (or dropped as a CoW dupe). LocalResolveDoFn keys on exact micros, so the two paths give different output for the same data.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergScanConfig.java:
##########
@@ -462,11 +481,76 @@ void validate(Table table) {
toSnapshotId);
}
+ if (fromSnapshotId != null) {
+ checkArgumentNotNull(
+ table.snapshot(fromSnapshotId),
+ error("configured starting snapshot does not exist: '%s'"),
+ fromSnapshotId);
+ }
+ if (toSnapshotId != null) {
+ checkArgumentNotNull(
+ table.snapshot(toSnapshotId),
+ error("configured end snapshot does not exist: '%s'"),
+ toSnapshotId);
+ }
+ if (fromSnapshotId != null && toSnapshotId != null) {
+ checkArgument(
+ SnapshotUtil.isAncestorOf(table, toSnapshotId, fromSnapshotId),
+ error("fromSnapshot '%s' is not an ancestor of toSnapshot '%s'"),
+ fromSnapshotId,
+ toSnapshotId);
+ }
+
if (getPollInterval() != null) {
checkArgument(
Boolean.TRUE.equals(getStreaming()),
error("'poll_interval_seconds' can only be set when streaming is true"));
}
+
+ @Nullable String watermarkColumn = getWatermarkColumn();
+ if (watermarkColumn != null) {
+ checkArgument(getUseCdc(), error("'watermark_column' is only supported in CDC mode"));
+ NestedField field = table.schema().findField(watermarkColumn);
+ checkArgument(
+ field != null, error("'watermark_column' refers to unknown column: %s"), watermarkColumn);
+ checkArgument(
+ field.isRequired(),
+ error("'watermark_column' needs to be a non-nullable column: %s"),
+ watermarkColumn);
+ checkArgument(
+ field.type().typeId() == TIMESTAMP || field.type().typeId() == LONG,
+ error("'watermark_column' must be a timestamp-typed column, but '%s' has type %s"),
+ watermarkColumn,
+ field.type().typeId());
+ checkArgumentNotNull(
+ getProjectedSchema().findField(watermarkColumn),
+ "'watermark_column' column should not be dropped.");
+ }
+
+ @Nullable String watermarkColumnTimeUnit = getWatermarkColumnTimeUnit();
+ if (watermarkColumnTimeUnit != null) {
+ checkArgument(
+ table
+ .schema()
+ .findField(
+ checkStateNotNull(
+ watermarkColumn,
+ "watermark_column_time_unit is configured without a specified watermark_column"))
+ .type()
+ .typeId()
+ == LONG,
+ error("watermark_column_time_unit is only applicable for LONG columns."));
+ try {
+ TimeUnit.valueOf(watermarkColumnTimeUnit.toUpperCase());
Review Comment:
The use of `String.toUpperCase()` without a locale can cause issues on JVMs set to locales like Turkish. Please consider using `toUpperCase(Locale.ENGLISH)` to ensure consistent behavior across all environments.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
Review Comment:
Consider noting that while records are buffered based on the 128MB compressed SPLIT_SIZE, the current implementation lacks an inflation factor for decoded data. With no safety check for the actual memory footprint in LocalResolveDoFn, a single process() call can exceed 500MB of heap usage. Furthermore, since the overlap bounds can be null when metrics are missing, the scanner defaults to buffering everything, which risks OOM errors.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/OverlapRange.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.util.Comparator;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.values.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Primary-key-projection and overlap-range comparison helper.
+ *
+ * <p>Used by {@link LocalResolveDoFn} and {@link ReadFromChangelogs} to decide whether a record's
+ * PK falls within an overlap of two opposing tasks. If so, the record needs to be compared with
+ * others to determine if it is part of an update pair.
+ */
+final class OverlapRange {
+ private final Schema recordIdSchema;
+ private final StructProjection recordIdProjection;
+ private final Comparator<StructLike> idComp;
+
+ private OverlapRange(
+ Schema recordIdSchema, StructProjection recordIdProjection, Comparator<StructLike> idComp) {
+ this.recordIdSchema = recordIdSchema;
+ this.recordIdProjection = recordIdProjection;
+ this.idComp = idComp;
+ }
+
+ static OverlapRange forScanConfig(IcebergScanConfig scanConfig) {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ StructProjection projection = StructProjection.create(fullSchema, scanConfig.recordIdSchema());
+ return new OverlapRange(
+ scanConfig.recordIdSchema(), projection, scanConfig.recordIdComparator());
+ }
+
+ StructProjection recordIdProjection() {
+ return recordIdProjection;
+ }
+
+ Schema recordIdSchema() {
+ return recordIdSchema;
+ }
+
+ /** Converts a Beam Row (overlap bound) back to an Iceberg {@link StructLike}. */
+ @Nullable
+ StructLike toStructLike(@Nullable Row beamBound) {
+ if (beamBound == null) {
+ return null;
+ }
+ return IcebergUtils.beamRowToIcebergRecord(recordIdSchema, beamBound);
Review Comment:
The current implementation performs a round-trip conversion from Iceberg micros to Beam DATETIME millis and back to micros, introducing a potential loss of precision up to 999µs. Records falling within this timing band may be misrouted between the direct-emit and resolution paths. This results in copy-on-write (CoW) no-ops appearing as spurious DELETE+INSERT pairs and genuine updates failing to resolve their pairings. We should ensure the bounds handling preserves original precision to prevent these classification errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3707830517
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
+ * <li>singletons remain {@code DELETE} or {@code INSERT}
+ * <li>matching delete+insert with identical non-PK fields are considered Copy-on-Write side
+ * effects and are dropped
+ * </ul>
+ *
+ * <p>General implementation:
+ *
+ * <ol>
+ * <li>Hash-index inserts by their non-PK field hash, for efficient Copy-on-Write detection.
+ * <li>Skip matching (delete, insert) pairs with identical non-PK columns. A CoW operation deletes
+ * and rewrites the whole file (minus some records that are actually marked for deletion).
+ * Unchanged records are no-ops and should not be mistaken for updates.
Review Comment:
A primary key can have identical (delete, insert) pairs in two scenarios:
1. There is a Copy-on-Write, where a new DataFile is added to replace an old DataFile now marked deleted. The new DataFile's rows are identical to the old DataFile, except for some missing rows that are interpreted to be deleted. The overlapping rows, on the surface, are identical (delete, insert) pairs.
2. Some records are inserted and then subsequently deleted in the same snapshot. Reading this produces identical (delete, insert) pairs. Our policy is to only look at the delta so we ignore them
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
+ * <li>singletons remain {@code DELETE} or {@code INSERT}
+ * <li>matching delete+insert with identical non-PK fields are considered Copy-on-Write side
+ * effects and are dropped
+ * </ul>
+ *
+ * <p>General implementation:
+ *
+ * <ol>
+ * <li>Hash-index inserts by their non-PK field hash, for efficient Copy-on-Write detection.
+ * <li>Skip matching (delete, insert) pairs with identical non-PK columns. A CoW operation deletes
+ * and rewrites the whole file (minus some records that are actually marked for deletion).
+ * Unchanged records are no-ops and should not be mistaken for updates.
+ * <li>Walk the remaining deletes and inserts, emitting matched pairs as {@link
+ * ValueKind#UPDATE_BEFORE} / {@link ValueKind#UPDATE_AFTER}.
+ * <li>Emit any unmatched extras as {@link ValueKind#DELETE} / {@link ValueKind#INSERT}.
+ * </ol>
+ */
+abstract class CdcResolver<T> {
+ /** Hashes the non-PK fields of an element. Used as the index for O(n+m) CoW deduplication. */
+ protected abstract int nonPkHash(T element);
+
+ /**
+ * Returns true if two records (already known to share a PK) share identical non-PK fields. Called
+ * only when the two elements collide in the {@link #nonPkHash} index, so the implementation can
+ * stay simple (linear scan of non-PK fields).
+ */
+ protected abstract boolean nonPkEquals(T delete, T insert);
+
+ /**
+ * Resolves a Primary Key group of deletes and inserts. Caller provides {@code emit} which decides
+ * how to materialize each output.
+ *
+ * <p>Both input lists are inspected in their given order.
+ */
+ final void resolve(List<T> deletes, List<T> inserts, BiConsumer<ValueKind, T> emit) {
+ boolean hasDeletes = !deletes.isEmpty();
+ boolean hasInserts = !inserts.isEmpty();
+
+ if (hasInserts && hasDeletes) {
+ // First, check if any (delete, insert) pairs are duplicates that should not be
+ // included in the output
+ boolean[] dupDeletes = new boolean[deletes.size()];
+ boolean[] dupInserts = new boolean[inserts.size()];
+
+ // Map hash to insert-indices
+ Map<Integer, List<Integer>> insertHashToIdx = new HashMap<>();
+ for (int insertIdx = 0; insertIdx < inserts.size(); insertIdx++) {
+ int insertHash = nonPkHash(inserts.get(insertIdx));
+ insertHashToIdx.computeIfAbsent(insertHash, k -> new ArrayList<>()).add(insertIdx);
+ }
+ for (int deleteIdx = 0; deleteIdx < deletes.size(); deleteIdx++) {
+ int deleteHash = nonPkHash(deletes.get(deleteIdx));
+ @Nullable List<Integer> candidates = insertHashToIdx.get(deleteHash);
+ if (candidates != null) {
+ // check if candidates are just duplicates (e.g. from CoW)
+ for (int idx = 0; idx < candidates.size(); idx++) {
+ int insertIdx = candidates.get(idx);
+ if (!dupInserts[insertIdx]
+ && nonPkEquals(deletes.get(deleteIdx), inserts.get(insertIdx))) {
+ // this (delete, insert) pair is a duplicate --> should be skipped
+ dupDeletes[deleteIdx] = true;
+ dupInserts[insertIdx] = true;
+ candidates.remove(idx);
+ break;
+ }
+ }
+ }
+ }
+
+ // Emit matched pairs as UPDATE_BEFORE / UPDATE_AFTER.
+ int d = 0;
+ int i = 0;
+ while (d < deletes.size() && i < inserts.size()) {
+ // skip duplicates
+ while (d < deletes.size() && dupDeletes[d]) {
+ d++;
+ }
+ while (i < inserts.size() && dupInserts[i]) {
+ i++;
+ }
+
+ if (d < deletes.size() && i < inserts.size()) {
+ emit.accept(ValueKind.UPDATE_BEFORE, deletes.get(d));
Review Comment:
Only when a PK group has multiple inserts or deletes. This would be unusual because it would mean one commit inserted or deleted a record multiple times. It's still possible though because Iceberg doesn't enforce uniqueness.
We unfortunately can't gather any further insight on intended record ordering within a commit, so pairing can be arbitrary. I've documented the case in the javadoc and now sort both sides by hash before pairing so the result is at least deterministic regardless of input order.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:
##########
@@ -471,120 +473,150 @@ private static Object getIcebergTimestampValue(Object beamValue, boolean shouldA
}
}
+ /** Converts a {@link StructLike} to a Beam {@link Row}. */
+ public static Row structToRow(Schema schema, StructLike struct) {
+ checkState(
+ schema.getFieldCount() == struct.size(),
+ "Struct of size %s does not match expected schema size %s",
+ struct.size(),
+ schema.getFieldCount());
+ Row.Builder rowBuilder = Row.withSchema(schema);
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ Schema.Field field = schema.getField(i);
+ @Nullable Object icebergValue = struct.get(i, Object.class);
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
/** Converts an Iceberg {@link Record} to a Beam {@link Row}. */
public static Row icebergRecordToBeamRow(Schema schema, Record record) {
Row.Builder rowBuilder = Row.withSchema(schema);
for (Schema.Field field : schema.getFields()) {
- boolean isNullable = field.getType().getNullable();
@Nullable Object icebergValue = record.getField(field.getName());
- if (icebergValue == null) {
- if (isNullable) {
- rowBuilder.addValue(null);
- continue;
- }
- throw new RuntimeException(
- String.format("Received null value for required field '%s'.", field.getName()));
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
+ private static void addIcebergValue(
+ Row.Builder rowBuilder, Schema.Field field, @Nullable Object icebergValue) {
+ boolean isNullable = field.getType().getNullable();
+ if (icebergValue == null) {
+ if (isNullable) {
+ rowBuilder.addValue(null);
+ return;
}
- switch (field.getType().getTypeName()) {
- case BYTE:
- case INT16:
- case INT32:
- case INT64:
- case DECIMAL: // Iceberg and Beam both use BigDecimal
- case FLOAT: // Iceberg and Beam both use float
- case DOUBLE: // Iceberg and Beam both use double
- case STRING: // Iceberg and Beam both use String
- case BOOLEAN: // Iceberg and Beam both use boolean
- rowBuilder.addValue(icebergValue);
- break;
- case ARRAY:
- checkState(
- icebergValue instanceof List,
- "Expected List type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
- Schema.FieldType collectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (collectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
- beamList =
- beamList.stream()
- .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
- .collect(Collectors.toList());
- }
- rowBuilder.addValue(beamList);
- break;
- case ITERABLE:
- checkState(
- icebergValue instanceof Iterable,
- "Expected Iterable type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
- Schema.FieldType iterableCollectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (iterableCollectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
- ImmutableList.Builder<Row> builder = ImmutableList.builder();
- for (Record v : (Iterable<@NonNull Record>) icebergValue) {
- builder.add(icebergRecordToBeamRow(innerSchema, v));
- }
- beamIterable = builder.build();
+ throw new RuntimeException(
+ String.format("Received null value for required field '%s'.", field.getName()));
+ }
+ switch (field.getType().getTypeName()) {
+ case BYTE:
+ case INT16:
+ case INT32:
+ case INT64:
+ case DECIMAL: // Iceberg and Beam both use BigDecimal
+ case FLOAT: // Iceberg and Beam both use float
+ case DOUBLE: // Iceberg and Beam both use double
+ case STRING: // Iceberg and Beam both use String
+ case BOOLEAN: // Iceberg and Beam both use boolean
+ rowBuilder.addValue(icebergValue);
+ break;
+ case ARRAY:
+ checkState(
+ icebergValue instanceof List,
+ "Expected List type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
+ Schema.FieldType collectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (collectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
+ beamList =
+ beamList.stream()
+ .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
+ .collect(Collectors.toList());
+ }
+ rowBuilder.addValue(beamList);
+ break;
+ case ITERABLE:
+ checkState(
+ icebergValue instanceof Iterable,
+ "Expected Iterable type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
+ Schema.FieldType iterableCollectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (iterableCollectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
+ ImmutableList.Builder<Row> builder = ImmutableList.builder();
+ for (Record v : (Iterable<@NonNull Record>) icebergValue) {
+ builder.add(icebergRecordToBeamRow(innerSchema, v));
}
- rowBuilder.addValue(beamIterable);
- break;
- case MAP:
- checkState(
- icebergValue instanceof Map,
- "Expected Map type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
- Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
- // recurse on struct types
- if (valueType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
- ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
- for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
- Record rec = ((Record) entry.getValue());
- newMap.put(
- checkStateNotNull(entry.getKey()),
- icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
- }
- beamMap = newMap.build();
+ beamIterable = builder.build();
+ }
+ rowBuilder.addValue(beamIterable);
+ break;
+ case MAP:
+ checkState(
+ icebergValue instanceof Map,
+ "Expected Map type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
+ Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
+ // recurse on struct types
+ if (valueType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
+ ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
+ for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
+ Record rec = ((Record) entry.getValue());
+ newMap.put(
+ checkStateNotNull(entry.getKey()),
+ icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
}
- rowBuilder.addValue(beamMap);
- break;
- case DATETIME:
- // Iceberg uses a long for micros.
- // Beam DATETIME uses joda's DateTime, which only supports millis,
- // so we do lose some precision here
- rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
- break;
- case BYTES:
- // Iceberg uses ByteBuffer; Beam uses byte[]
- rowBuilder.addValue(((ByteBuffer) icebergValue).array());
- break;
- case ROW:
- Record nestedRecord = (Record) icebergValue;
- Schema nestedSchema =
- checkArgumentNotNull(
- field.getType().getRowSchema(),
- "Corrupted schema: Row type did not have associated nested schema.");
- rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
- break;
- case LOGICAL_TYPE:
- rowBuilder.addValue(getLogicalTypeValue(icebergValue, field.getType()));
- break;
- default:
+ beamMap = newMap.build();
+ }
+ rowBuilder.addValue(beamMap);
+ break;
+ case DATETIME:
+ // Iceberg uses a long for micros.
+ // Beam DATETIME uses joda's DateTime, which only supports millis,
+ // so we do lose some precision here
Review Comment:
Can't really fix this, it's a known shortcoming of Joda which is what Beam DATETIME uses.
The workaround is to use the new Timestamp.MICROS logical type (backed by Java time), which has just been made the new default (https://github.com/apache/beam/pull/39344).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3565660616
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:
##########
@@ -469,120 +470,141 @@ private static Object getIcebergTimestampValue(Object beamValue, boolean shouldA
}
}
+ /** Converts a {@link StructLike} to a Beam {@link Row}. */
+ public static Row structToRow(Schema schema, StructLike struct) {
+ checkState(
+ schema.getFieldCount() == struct.size(),
+ "Struct of size %s does not match expected schema size %s",
+ struct.size(),
+ schema.getFieldCount());
+ Row.Builder rowBuilder = Row.withSchema(schema);
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ Schema.Field field = schema.getField(i);
+ @Nullable Object icebergValue = struct.get(i, Object.class);
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
/** Converts an Iceberg {@link Record} to a Beam {@link Row}. */
public static Row icebergRecordToBeamRow(Schema schema, Record record) {
Row.Builder rowBuilder = Row.withSchema(schema);
for (Schema.Field field : schema.getFields()) {
- boolean isNullable = field.getType().getNullable();
@Nullable Object icebergValue = record.getField(field.getName());
- if (icebergValue == null) {
- if (isNullable) {
- rowBuilder.addValue(null);
- continue;
- }
- throw new RuntimeException(
- String.format("Received null value for required field '%s'.", field.getName()));
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
+ private static void addIcebergValue(
+ Row.Builder rowBuilder, Schema.Field field, @Nullable Object icebergValue) {
+ boolean isNullable = field.getType().getNullable();
+ if (icebergValue == null) {
+ if (isNullable) {
+ rowBuilder.addValue(null);
+ return;
}
- switch (field.getType().getTypeName()) {
- case BYTE:
- case INT16:
- case INT32:
- case INT64:
- case DECIMAL: // Iceberg and Beam both use BigDecimal
- case FLOAT: // Iceberg and Beam both use float
- case DOUBLE: // Iceberg and Beam both use double
- case STRING: // Iceberg and Beam both use String
- case BOOLEAN: // Iceberg and Beam both use boolean
- rowBuilder.addValue(icebergValue);
- break;
- case ARRAY:
- checkState(
- icebergValue instanceof List,
- "Expected List type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
- Schema.FieldType collectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (collectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
- beamList =
- beamList.stream()
- .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
- .collect(Collectors.toList());
- }
- rowBuilder.addValue(beamList);
- break;
- case ITERABLE:
- checkState(
- icebergValue instanceof Iterable,
- "Expected Iterable type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
- Schema.FieldType iterableCollectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (iterableCollectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
- ImmutableList.Builder<Row> builder = ImmutableList.builder();
- for (Record v : (Iterable<@NonNull Record>) icebergValue) {
- builder.add(icebergRecordToBeamRow(innerSchema, v));
- }
- beamIterable = builder.build();
+ throw new RuntimeException(
+ String.format("Received null value for required field '%s'.", field.getName()));
+ }
+ switch (field.getType().getTypeName()) {
+ case BYTE:
+ case INT16:
+ case INT32:
+ case INT64:
+ case DECIMAL: // Iceberg and Beam both use BigDecimal
+ case FLOAT: // Iceberg and Beam both use float
+ case DOUBLE: // Iceberg and Beam both use double
+ case STRING: // Iceberg and Beam both use String
+ case BOOLEAN: // Iceberg and Beam both use boolean
+ rowBuilder.addValue(icebergValue);
+ break;
+ case ARRAY:
+ checkState(
+ icebergValue instanceof List,
+ "Expected List type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
+ Schema.FieldType collectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (collectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
+ beamList =
+ beamList.stream()
+ .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
+ .collect(Collectors.toList());
+ }
+ rowBuilder.addValue(beamList);
+ break;
+ case ITERABLE:
+ checkState(
+ icebergValue instanceof Iterable,
+ "Expected Iterable type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
+ Schema.FieldType iterableCollectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (iterableCollectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
+ ImmutableList.Builder<Row> builder = ImmutableList.builder();
+ for (Record v : (Iterable<@NonNull Record>) icebergValue) {
+ builder.add(icebergRecordToBeamRow(innerSchema, v));
}
- rowBuilder.addValue(beamIterable);
- break;
- case MAP:
- checkState(
- icebergValue instanceof Map,
- "Expected Map type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
- Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
- // recurse on struct types
- if (valueType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
- ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
- for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
- Record rec = ((Record) entry.getValue());
- newMap.put(
- checkStateNotNull(entry.getKey()),
- icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
- }
- beamMap = newMap.build();
+ beamIterable = builder.build();
+ }
+ rowBuilder.addValue(beamIterable);
+ break;
+ case MAP:
+ checkState(
+ icebergValue instanceof Map,
+ "Expected Map type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
+ Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
+ // recurse on struct types
+ if (valueType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
+ ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
+ for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
+ Record rec = ((Record) entry.getValue());
+ newMap.put(
+ checkStateNotNull(entry.getKey()),
+ icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
}
- rowBuilder.addValue(beamMap);
- break;
- case DATETIME:
- // Iceberg uses a long for micros.
- // Beam DATETIME uses joda's DateTime, which only supports millis,
- // so we do lose some precision here
- rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
- break;
- case BYTES:
- // Iceberg uses ByteBuffer; Beam uses byte[]
- rowBuilder.addValue(((ByteBuffer) icebergValue).array());
- break;
- case ROW:
- Record nestedRecord = (Record) icebergValue;
- Schema nestedSchema =
- checkArgumentNotNull(
- field.getType().getRowSchema(),
- "Corrupted schema: Row type did not have associated nested schema.");
- rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
- break;
- case LOGICAL_TYPE:
- rowBuilder.addValue(getLogicalTypeValue(icebergValue, field.getType()));
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported Beam type: " + field.getType().getTypeName());
- }
+ beamMap = newMap.build();
+ }
+ rowBuilder.addValue(beamMap);
+ break;
+ case DATETIME:
+ // Iceberg uses a long for micros.
+ // Beam DATETIME uses joda's DateTime, which only supports millis,
+ // so we do lose some precision here
+ rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
+ break;
+ case BYTES:
+ // Iceberg uses ByteBuffer; Beam uses byte[]
+ rowBuilder.addValue(((ByteBuffer) icebergValue).array());
+ break;
+ case ROW:
+ Record nestedRecord = (Record) icebergValue;
+ Schema nestedSchema =
+ checkArgumentNotNull(
+ field.getType().getRowSchema(),
+ "Corrupted schema: Row type did not have associated nested schema.");
+ rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
Review Comment:
Nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3589764832
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
+
+ @Nullable StructLike overlapLower = ovl.toStructLike(descriptor.getOverlapLower());
+ @Nullable StructLike overlapUpper = ovl.toStructLike(descriptor.getOverlapUpper());
+ for (SerializableChangelogTask task : element.getValue()) {
+ readAndRoute(descriptor, task, table, overlapLower, overlapUpper, pkGroups, out);
+ }
+
+ resolveAndEmit(
+ descriptor,
+ pkGroups,
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), table.schema()),
+ out);
+ }
+
+ /**
+ * Processes a {@link SerializableChangelogTask} and routes each record.
+ *
+ * <ul>
+ * <li>Out of overlap: emit directly
+ * <li>Inside overlap: stash in {@code pkGroups} to resolve in {@link #resolveAndEmit}
+ * </ul>
+ */
+ private void readAndRoute(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ Table table,
+ @Nullable StructLike overlapLower,
+ @Nullable StructLike overlapUpper,
+ StructLikeMap<PkGroup> pkGroups,
+ OutputReceiver<Row> out)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, false)) {
+ for (Record rec : records) {
+ if (ovl.contains(rec, overlapLower, overlapUpper)) { // needs resolution
+ StructLike pk = StructLikeUtil.copy(ovl.recordIdProjection());
+ PkGroup group = pkGroups.computeIfAbsent(pk, k -> new PkGroup());
+ if (isInsert) {
+ group.inserts.add(rec);
+ } else {
+ group.deletes.add(rec);
+ }
+ } else { // safe to emit directly
+ emit(descriptor, rec, isInsert ? ValueKind.INSERT : ValueKind.DELETE, out);
+ }
+ }
+ }
+ }
+
+ /** Resolves each PK group using {@link CdcResolver}. */
+ private void resolveAndEmit(
+ ChangelogDescriptor descriptor,
+ StructLikeMap<PkGroup> pkGroups,
+ Schema fullSchema,
+ OutputReceiver<Row> out) {
+ CdcResolver<Record> resolver = new RecordResolver(checkStateNotNull(nonPkFields), fullSchema);
+ for (PkGroup group : pkGroups.values()) {
+ resolver.resolve(
+ group.deletes,
+ group.inserts,
+ (kind, rec) -> {
+ emit(descriptor, rec, kind, out);
+ });
+ }
+ }
+
+ /** Resolver specialization that hashes Iceberg Record non-PK fields. */
+ private static final class RecordResolver extends CdcResolver<Record> {
+ private final List<Types.NestedField> nonPkFields;
+ private final Comparator<StructLike> nonPkComparator;
+ private final StructProjection left;
+ private final StructProjection right;
+
+ RecordResolver(List<Types.NestedField> nonPkFields, Schema recSchema) {
+ this.nonPkFields = nonPkFields;
+ Set<Integer> nonPkFieldIds =
+ nonPkFields.stream().map(Types.NestedField::fieldId).collect(Collectors.toSet());
+ this.left = StructProjection.create(recSchema, nonPkFieldIds);
+ this.right = StructProjection.create(recSchema, nonPkFieldIds);
+ this.nonPkComparator =
+ Comparators.forType(TypeUtil.select(recSchema, nonPkFieldIds).asStruct());
+ }
+
+ @Override
+ protected int nonPkHash(Record rec) {
+ int hash = 1;
+ for (Types.NestedField field : nonPkFields) {
+ hash = 31 * hash + Objects.hashCode(rec.getField(field.name()));
Review Comment:
> Can we derive both hash and equals from one mechanism
We could build a single content-based key per overlapping record and do a equals/hashCode for each lookup, but that would be less performant
Correct that the old code was problematic for `fixed` types cuz Objects.hashCode() would be identity based for `byte[]`. I covered this by making the hash logic array-aware. This way, we only check deep-equality on a hash collision. I also added tests that a CoW pair with a non-PK `fixed` column is recognized and dropped.
I'd rather keep it this way cuz it's more performant, but let me know what you think.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
Review Comment:
Good point. I opened a PR to return a java Instant instead (preserves micros) for timestamptz: #39344
It's been a latent piece of work I've been meaning to get in, and now is a good time. Should take care of this point
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/OverlapRange.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.util.Comparator;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.values.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Primary-key-projection and overlap-range comparison helper.
+ *
+ * <p>Used by {@link LocalResolveDoFn} and {@link ReadFromChangelogs} to decide whether a record's
+ * PK falls within an overlap of two opposing tasks. If so, the record needs to be compared with
+ * others to determine if it is part of an update pair.
+ */
+final class OverlapRange {
+ private final Schema recordIdSchema;
+ private final StructProjection recordIdProjection;
+ private final Comparator<StructLike> idComp;
+
+ private OverlapRange(
+ Schema recordIdSchema, StructProjection recordIdProjection, Comparator<StructLike> idComp) {
+ this.recordIdSchema = recordIdSchema;
+ this.recordIdProjection = recordIdProjection;
+ this.idComp = idComp;
+ }
+
+ static OverlapRange forScanConfig(IcebergScanConfig scanConfig) {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ StructProjection projection = StructProjection.create(fullSchema, scanConfig.recordIdSchema());
+ return new OverlapRange(
+ scanConfig.recordIdSchema(), projection, scanConfig.recordIdComparator());
+ }
+
+ StructProjection recordIdProjection() {
+ return recordIdProjection;
+ }
+
+ Schema recordIdSchema() {
+ return recordIdSchema;
+ }
+
+ /** Converts a Beam Row (overlap bound) back to an Iceberg {@link StructLike}. */
+ @Nullable
+ StructLike toStructLike(@Nullable Row beamBound) {
+ if (beamBound == null) {
+ return null;
+ }
+ return IcebergUtils.beamRowToIcebergRecord(recordIdSchema, beamBound);
Review Comment:
Similar to the previous comment, I'm routing `timestmptz` to java Instant to preserve micros: #39344
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
+ outputReceiver
+ .get(taggedOutput)
+ .builder(
+ KV.of(
+ CdcRowDescriptor.builder()
+ .setCommitSnapshotId(commitSnapshotId)
+ .setSnapshotSequenceNumber(commitSnapshotSequenceNumber)
+ .setPrimaryKey(pk)
+ .build(),
+ row))
+ .setValueKind(kind)
+ .output();
+
+ } else {
+ // outside overlap -- get projected record and output
+ StructLike projected = outputProjector().wrap(rec);
+ Row row = structToRow(projectedBeamRowSchema, projected);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ }
+ }
+ }
+
+ trackMetrics(task.getType());
+ }
+
+ private StructProjection outputProjector() {
+ if (outputProjector == null) {
+ outputProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+ return outputProjector;
+ }
+
+ private StructProjection pkProjector() {
+ if (pkProjector == null) {
+ pkProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ scanConfig.recordIdSchema());
+ }
+ return pkProjector;
+ }
+
+ private void trackMetrics(SerializableChangelogTask.Type type) {
+ switch (type) {
+ case ADDED_ROWS:
+ numAddedRowsScanTasksCompleted.inc();
+ break;
+ case DELETED_ROWS:
+ numDeletedRowsScanTasksCompleted.inc();
+ break;
+ case DELETED_FILE:
+ numDeletedDataFileScanTasksCompleted.inc();
+ break;
+ }
+ }
+
+ private String getKind(SerializableChangelogTask.Type taskType) {
+ switch (taskType) {
+ case ADDED_ROWS:
+ return "INSERT";
+ case DELETED_ROWS:
+ return "DELETE";
+ case DELETED_FILE:
+ default:
+ return "DELETE-DF";
+ }
+ }
+
+ private static final int COMPRESSED_TO_DECODED_BYTES_ESTIMATE = 4;
+
+ @GetSize
+ public double getSize(
Review Comment:
Good catch, switched to sum by `getLength()` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3565653078
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:
##########
@@ -469,120 +470,141 @@ private static Object getIcebergTimestampValue(Object beamValue, boolean shouldA
}
}
+ /** Converts a {@link StructLike} to a Beam {@link Row}. */
+ public static Row structToRow(Schema schema, StructLike struct) {
+ checkState(
+ schema.getFieldCount() == struct.size(),
+ "Struct of size %s does not match expected schema size %s",
+ struct.size(),
+ schema.getFieldCount());
+ Row.Builder rowBuilder = Row.withSchema(schema);
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ Schema.Field field = schema.getField(i);
+ @Nullable Object icebergValue = struct.get(i, Object.class);
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
/** Converts an Iceberg {@link Record} to a Beam {@link Row}. */
public static Row icebergRecordToBeamRow(Schema schema, Record record) {
Row.Builder rowBuilder = Row.withSchema(schema);
for (Schema.Field field : schema.getFields()) {
- boolean isNullable = field.getType().getNullable();
@Nullable Object icebergValue = record.getField(field.getName());
- if (icebergValue == null) {
- if (isNullable) {
- rowBuilder.addValue(null);
- continue;
- }
- throw new RuntimeException(
- String.format("Received null value for required field '%s'.", field.getName()));
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
+ private static void addIcebergValue(
+ Row.Builder rowBuilder, Schema.Field field, @Nullable Object icebergValue) {
+ boolean isNullable = field.getType().getNullable();
+ if (icebergValue == null) {
+ if (isNullable) {
+ rowBuilder.addValue(null);
+ return;
}
- switch (field.getType().getTypeName()) {
- case BYTE:
- case INT16:
- case INT32:
- case INT64:
- case DECIMAL: // Iceberg and Beam both use BigDecimal
- case FLOAT: // Iceberg and Beam both use float
- case DOUBLE: // Iceberg and Beam both use double
- case STRING: // Iceberg and Beam both use String
- case BOOLEAN: // Iceberg and Beam both use boolean
- rowBuilder.addValue(icebergValue);
- break;
- case ARRAY:
- checkState(
- icebergValue instanceof List,
- "Expected List type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
- Schema.FieldType collectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (collectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
- beamList =
- beamList.stream()
- .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
- .collect(Collectors.toList());
- }
- rowBuilder.addValue(beamList);
- break;
- case ITERABLE:
- checkState(
- icebergValue instanceof Iterable,
- "Expected Iterable type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
- Schema.FieldType iterableCollectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (iterableCollectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
- ImmutableList.Builder<Row> builder = ImmutableList.builder();
- for (Record v : (Iterable<@NonNull Record>) icebergValue) {
- builder.add(icebergRecordToBeamRow(innerSchema, v));
- }
- beamIterable = builder.build();
+ throw new RuntimeException(
+ String.format("Received null value for required field '%s'.", field.getName()));
+ }
+ switch (field.getType().getTypeName()) {
+ case BYTE:
+ case INT16:
+ case INT32:
+ case INT64:
+ case DECIMAL: // Iceberg and Beam both use BigDecimal
+ case FLOAT: // Iceberg and Beam both use float
+ case DOUBLE: // Iceberg and Beam both use double
+ case STRING: // Iceberg and Beam both use String
+ case BOOLEAN: // Iceberg and Beam both use boolean
+ rowBuilder.addValue(icebergValue);
+ break;
+ case ARRAY:
+ checkState(
+ icebergValue instanceof List,
+ "Expected List type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
+ Schema.FieldType collectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (collectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
+ beamList =
+ beamList.stream()
+ .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
+ .collect(Collectors.toList());
+ }
+ rowBuilder.addValue(beamList);
+ break;
+ case ITERABLE:
+ checkState(
+ icebergValue instanceof Iterable,
+ "Expected Iterable type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
+ Schema.FieldType iterableCollectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (iterableCollectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
+ ImmutableList.Builder<Row> builder = ImmutableList.builder();
+ for (Record v : (Iterable<@NonNull Record>) icebergValue) {
+ builder.add(icebergRecordToBeamRow(innerSchema, v));
}
- rowBuilder.addValue(beamIterable);
- break;
- case MAP:
- checkState(
- icebergValue instanceof Map,
- "Expected Map type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
- Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
- // recurse on struct types
- if (valueType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
- ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
- for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
- Record rec = ((Record) entry.getValue());
- newMap.put(
- checkStateNotNull(entry.getKey()),
- icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
- }
- beamMap = newMap.build();
+ beamIterable = builder.build();
+ }
+ rowBuilder.addValue(beamIterable);
+ break;
+ case MAP:
+ checkState(
+ icebergValue instanceof Map,
+ "Expected Map type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
+ Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
+ // recurse on struct types
+ if (valueType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
+ ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
+ for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
+ Record rec = ((Record) entry.getValue());
+ newMap.put(
+ checkStateNotNull(entry.getKey()),
+ icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
}
- rowBuilder.addValue(beamMap);
- break;
- case DATETIME:
- // Iceberg uses a long for micros.
- // Beam DATETIME uses joda's DateTime, which only supports millis,
- // so we do lose some precision here
- rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
- break;
- case BYTES:
- // Iceberg uses ByteBuffer; Beam uses byte[]
- rowBuilder.addValue(((ByteBuffer) icebergValue).array());
- break;
- case ROW:
- Record nestedRecord = (Record) icebergValue;
- Schema nestedSchema =
- checkArgumentNotNull(
- field.getType().getRowSchema(),
- "Corrupted schema: Row type did not have associated nested schema.");
- rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
- break;
- case LOGICAL_TYPE:
- rowBuilder.addValue(getLogicalTypeValue(icebergValue, field.getType()));
- break;
- default:
- throw new UnsupportedOperationException(
- "Unsupported Beam type: " + field.getType().getTypeName());
- }
+ beamMap = newMap.build();
+ }
+ rowBuilder.addValue(beamMap);
+ break;
+ case DATETIME:
+ // Iceberg uses a long for micros.
+ // Beam DATETIME uses joda's DateTime, which only supports millis,
+ // so we do lose some precision here
+ rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
+ break;
+ case BYTES:
+ // Iceberg uses ByteBuffer; Beam uses byte[]
+ rowBuilder.addValue(((ByteBuffer) icebergValue).array());
+ break;
+ case ROW:
+ Record nestedRecord = (Record) icebergValue;
+ Schema nestedSchema =
+ checkArgumentNotNull(
+ field.getType().getRowSchema(),
+ "Corrupted schema: Row type did not have associated nested schema.");
+ rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
Review Comment:

Casting `icebergValue` directly to `Record` will throw a `ClassCastException` if the nested object is a `StructLike` (such as a `StructProjection`) but not a `Record`. Since `structToRow` can be called with `StructProjection` containing nested structs, we should check the type of `icebergValue` and handle both `Record` and `StructLike` appropriately.
```suggestion
Schema nestedSchema =
checkArgumentNotNull(
field.getType().getRowSchema(),
"Corrupted schema: Row type did not have associated nested schema.");
if (icebergValue instanceof Record) {
rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, (Record) icebergValue));
} else if (icebergValue instanceof StructLike) {
rowBuilder.addValue(structToRow(nestedSchema, (StructLike) icebergValue));
} else {
throw new UnsupportedOperationException("Unsupported row type: " + icebergValue.getClass());
}
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,493 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
+ outputReceiver
+ .get(taggedOutput)
+ .builder(
+ KV.of(
+ CdcRowDescriptor.builder()
+ .setCommitSnapshotId(commitSnapshotId)
+ .setSnapshotSequenceNumber(commitSnapshotSequenceNumber)
+ .setPrimaryKey(pk)
+ .build(),
+ row))
+ .setValueKind(kind)
+ .output();
+
+ } else {
+ // outside overlap -- get projected record and output
+ StructLike projected = outputProjector().wrap(rec);
+ Row row = structToRow(projectedBeamRowSchema, projected);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ }
+ }
+ }
+
+ trackMetrics(task.getType());
+ }
+
+ private StructProjection outputProjector() {
+ if (outputProjector == null) {
+ outputProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+ return outputProjector;
+ }
+
+ private StructProjection pkProjector() {
+ if (pkProjector == null) {
+ pkProjector =
+ StructProjection.create(
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema(),
+ scanConfig.recordIdSchema());
Review Comment:

The `pkProjector` is created using `table.schema()`, but the records it wraps have `fullReadSchema` (which includes metadata columns). This schema mismatch will cause a position mismatch when `StructProjection` attempts to access fields of the wrapped record. We should construct the source schema for `pkProjector` using `CdcOutputUtils.readSchemaWithRowMetadata` to match the schema of the records being processed, consistent with `outputProjector()`.
```java
StructProjection.create(
CdcOutputUtils.readSchemaWithRowMetadata(
scanConfig.getMetadataColumns(),
TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
.schema()),
scanConfig.recordIdSchema());
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
+
+ @Nullable StructLike overlapLower = ovl.toStructLike(descriptor.getOverlapLower());
+ @Nullable StructLike overlapUpper = ovl.toStructLike(descriptor.getOverlapUpper());
+ for (SerializableChangelogTask task : element.getValue()) {
+ readAndRoute(descriptor, task, table, overlapLower, overlapUpper, pkGroups, out);
+ }
+
+ resolveAndEmit(descriptor, pkGroups, table.schema(), out);
Review Comment:

The records in `pkGroups` are read with `projected = false`, meaning they have `fullReadSchema` (which includes metadata columns). However, `resolveAndEmit` is called with `table.schema()` (which does not include metadata columns). This schema mismatch will cause a position mismatch when `RecordResolver`'s `StructProjection` attempts to compare non-PK fields. We should pass the schema with metadata columns to `resolveAndEmit`.
```java
resolveAndEmit(
descriptor,
pkGroups,
CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), table.schema()),
out);
```
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/OverlapRange.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.util.Comparator;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.values.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Primary-key-projection and overlap-range comparison helper.
+ *
+ * <p>Used by {@link LocalResolveDoFn} and {@link ReadFromChangelogs} to decide whether a record's
+ * PK falls within an overlap of two opposing tasks. If so, the record needs to be compared with
+ * others to determine if it is part of an update pair.
+ */
+final class OverlapRange {
+ private final Schema recordIdSchema;
+ private final StructProjection recordIdProjection;
+ private final Comparator<StructLike> idComp;
+
+ private OverlapRange(
+ Schema recordIdSchema, StructProjection recordIdProjection, Comparator<StructLike> idComp) {
+ this.recordIdSchema = recordIdSchema;
+ this.recordIdProjection = recordIdProjection;
+ this.idComp = idComp;
+ }
+
+ static OverlapRange forScanConfig(IcebergScanConfig scanConfig) {
+ Schema fullSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ StructProjection projection = StructProjection.create(fullSchema, scanConfig.recordIdSchema());
Review Comment:

The records evaluated by `contains` have `fullReadSchema` (which includes metadata columns) because they are read with `projected = false`. However, `recordIdProjection` is created using `table.schema()` (which does not include metadata columns). This schema mismatch will cause a position mismatch when `StructProjection` attempts to access fields of the wrapped record. We should construct `fullSchema` using `CdcOutputUtils.readSchemaWithRowMetadata` to include the metadata columns.
```suggestion
Schema tableSchema =
TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
Schema fullSchema =
CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
StructProjection projection = StructProjection.create(fullSchema, scanConfig.recordIdSchema());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38837:
URL: https://github.com/apache/beam/pull/38837#issuecomment-4949918480
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces the core infrastructure for reading and resolving Iceberg CDC changelogs within Apache Beam. It implements a routing mechanism that distinguishes between unidirectional and bidirectional changelog tasks, ensuring that records are either emitted directly or routed for update resolution based on primary key overlap. By centralizing the reconciliation logic, this change enables consistent handling of CDC data, including the identification of update pairs and the filtering of redundant operations.
### Highlights
* **Read Transform Implementation**: Added ReadFromChangelogs transform to process and route Iceberg changelog tasks based on their type and primary key overlap.
* **CDC Resolution Logic**: Implemented CdcResolver to reconcile insert and delete operations into update events, including support for Copy-on-Write deduplication.
* **In-Memory Processing**: Added LocalResolveDoFn to efficiently resolve small bidirectional changelog batches entirely in memory.
* **Configuration Enhancements**: Updated IcebergScanConfig to support watermark columns and added robust validation for CDC-specific configurations.
<details>
<summary><b>New Features</b></summary>
🧠 You can now enable **Memory** (public preview) to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #38837:
URL: https://github.com/apache/beam/pull/38837#issuecomment-4949986883
R: @talatuyarer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #38837:
URL: https://github.com/apache/beam/pull/38837#issuecomment-5172498720
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/38837?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 58.15%. Comparing base ([`2621e9e`](https://app.codecov.io/gh/apache/beam/commit/2621e9e047c0ae168e97cc266859b5fa4097d9bb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b6f561a`](https://app.codecov.io/gh/apache/beam/commit/b6f561ab2c3dbf244d197f167f9cf2a99f46c848?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 18 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #38837 +/- ##
============================================
- Coverage 58.17% 58.15% -0.02%
- Complexity 13086 13087 +1
============================================
Files 2521 2521
Lines 264569 264573 +4
Branches 10788 10788
============================================
- Hits 153902 153872 -30
- Misses 104900 104929 +29
- Partials 5767 5772 +5
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/38837/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [java](https://app.codecov.io/gh/apache/beam/pull/38837/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.24% <ø> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/38837?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3589766223
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergScanConfig.java:
##########
@@ -462,11 +481,76 @@ void validate(Table table) {
toSnapshotId);
}
+ if (fromSnapshotId != null) {
+ checkArgumentNotNull(
+ table.snapshot(fromSnapshotId),
+ error("configured starting snapshot does not exist: '%s'"),
+ fromSnapshotId);
+ }
+ if (toSnapshotId != null) {
+ checkArgumentNotNull(
+ table.snapshot(toSnapshotId),
+ error("configured end snapshot does not exist: '%s'"),
+ toSnapshotId);
+ }
+ if (fromSnapshotId != null && toSnapshotId != null) {
+ checkArgument(
+ SnapshotUtil.isAncestorOf(table, toSnapshotId, fromSnapshotId),
+ error("fromSnapshot '%s' is not an ancestor of toSnapshot '%s'"),
+ fromSnapshotId,
+ toSnapshotId);
+ }
+
if (getPollInterval() != null) {
checkArgument(
Boolean.TRUE.equals(getStreaming()),
error("'poll_interval_seconds' can only be set when streaming is true"));
}
+
+ @Nullable String watermarkColumn = getWatermarkColumn();
+ if (watermarkColumn != null) {
+ checkArgument(getUseCdc(), error("'watermark_column' is only supported in CDC mode"));
+ NestedField field = table.schema().findField(watermarkColumn);
+ checkArgument(
+ field != null, error("'watermark_column' refers to unknown column: %s"), watermarkColumn);
+ checkArgument(
+ field.isRequired(),
+ error("'watermark_column' needs to be a non-nullable column: %s"),
+ watermarkColumn);
+ checkArgument(
+ field.type().typeId() == TIMESTAMP || field.type().typeId() == LONG,
+ error("'watermark_column' must be a timestamp-typed column, but '%s' has type %s"),
+ watermarkColumn,
+ field.type().typeId());
+ checkArgumentNotNull(
+ getProjectedSchema().findField(watermarkColumn),
+ "'watermark_column' column should not be dropped.");
+ }
+
+ @Nullable String watermarkColumnTimeUnit = getWatermarkColumnTimeUnit();
+ if (watermarkColumnTimeUnit != null) {
+ checkArgument(
+ table
+ .schema()
+ .findField(
+ checkStateNotNull(
+ watermarkColumn,
+ "watermark_column_time_unit is configured without a specified watermark_column"))
+ .type()
+ .typeId()
+ == LONG,
+ error("watermark_column_time_unit is only applicable for LONG columns."));
+ try {
+ TimeUnit.valueOf(watermarkColumnTimeUnit.toUpperCase());
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3707857493
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/OverlapRange.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.util.Comparator;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.values.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Primary-key-projection and overlap-range comparison helper.
+ *
+ * <p>Used by {@link LocalResolveDoFn} and {@link ReadFromChangelogs} to decide whether a record's
Review Comment:
It's different from split ranges because these are actual data values, not partitions. If we make the range open on one end, we'd miss a collision and accidentally process an update pair as separate delete + insert.
I added some language to clarify. We already have a few boundary tests in `OverlapRangeTest` but I added a couple more to cover edge cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3678150399
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java:
##########
@@ -70,6 +70,7 @@
*/
public final class CdcReadUtils {
private static final Logger LOG = LoggerFactory.getLogger(CdcReadUtils.class);
+ static final int COMPRESSED_TO_DECODED_BYTES_ESTIMATE = 4;
Review Comment:
Probably add a small docstring for the non-private constant.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
Review Comment:
(delete, insert) pairs with the same key ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
+ * <li>singletons remain {@code DELETE} or {@code INSERT}
+ * <li>matching delete+insert with identical non-PK fields are considered Copy-on-Write side
+ * effects and are dropped
+ * </ul>
+ *
+ * <p>General implementation:
+ *
+ * <ol>
+ * <li>Hash-index inserts by their non-PK field hash, for efficient Copy-on-Write detection.
+ * <li>Skip matching (delete, insert) pairs with identical non-PK columns. A CoW operation deletes
+ * and rewrites the whole file (minus some records that are actually marked for deletion).
+ * Unchanged records are no-ops and should not be mistaken for updates.
+ * <li>Walk the remaining deletes and inserts, emitting matched pairs as {@link
+ * ValueKind#UPDATE_BEFORE} / {@link ValueKind#UPDATE_AFTER}.
+ * <li>Emit any unmatched extras as {@link ValueKind#DELETE} / {@link ValueKind#INSERT}.
+ * </ol>
+ */
+abstract class CdcResolver<T> {
+ /** Hashes the non-PK fields of an element. Used as the index for O(n+m) CoW deduplication. */
+ protected abstract int nonPkHash(T element);
+
+ /**
+ * Returns true if two records (already known to share a PK) share identical non-PK fields. Called
+ * only when the two elements collide in the {@link #nonPkHash} index, so the implementation can
+ * stay simple (linear scan of non-PK fields).
+ */
+ protected abstract boolean nonPkEquals(T delete, T insert);
+
+ /**
+ * Resolves a Primary Key group of deletes and inserts. Caller provides {@code emit} which decides
+ * how to materialize each output.
+ *
+ * <p>Both input lists are inspected in their given order.
+ */
+ final void resolve(List<T> deletes, List<T> inserts, BiConsumer<ValueKind, T> emit) {
+ boolean hasDeletes = !deletes.isEmpty();
+ boolean hasInserts = !inserts.isEmpty();
+
+ if (hasInserts && hasDeletes) {
+ // First, check if any (delete, insert) pairs are duplicates that should not be
+ // included in the output
+ boolean[] dupDeletes = new boolean[deletes.size()];
+ boolean[] dupInserts = new boolean[inserts.size()];
+
+ // Map hash to insert-indices
+ Map<Integer, List<Integer>> insertHashToIdx = new HashMap<>();
+ for (int insertIdx = 0; insertIdx < inserts.size(); insertIdx++) {
+ int insertHash = nonPkHash(inserts.get(insertIdx));
+ insertHashToIdx.computeIfAbsent(insertHash, k -> new ArrayList<>()).add(insertIdx);
+ }
+ for (int deleteIdx = 0; deleteIdx < deletes.size(); deleteIdx++) {
+ int deleteHash = nonPkHash(deletes.get(deleteIdx));
+ @Nullable List<Integer> candidates = insertHashToIdx.get(deleteHash);
+ if (candidates != null) {
+ // check if candidates are just duplicates (e.g. from CoW)
+ for (int idx = 0; idx < candidates.size(); idx++) {
+ int insertIdx = candidates.get(idx);
+ if (!dupInserts[insertIdx]
+ && nonPkEquals(deletes.get(deleteIdx), inserts.get(insertIdx))) {
+ // this (delete, insert) pair is a duplicate --> should be skipped
+ dupDeletes[deleteIdx] = true;
+ dupInserts[insertIdx] = true;
+ candidates.remove(idx);
+ break;
+ }
+ }
+ }
+ }
+
+ // Emit matched pairs as UPDATE_BEFORE / UPDATE_AFTER.
+ int d = 0;
+ int i = 0;
+ while (d < deletes.size() && i < inserts.size()) {
+ // skip duplicates
+ while (d < deletes.size() && dupDeletes[d]) {
+ d++;
+ }
+ while (i < inserts.size() && dupInserts[i]) {
+ i++;
+ }
+
+ if (d < deletes.size() && i < inserts.size()) {
+ emit.accept(ValueKind.UPDATE_BEFORE, deletes.get(d));
Review Comment:
Seems like deletes and inserts have to be in a particular order to match here correctly ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
+ * <li>singletons remain {@code DELETE} or {@code INSERT}
+ * <li>matching delete+insert with identical non-PK fields are considered Copy-on-Write side
+ * effects and are dropped
+ * </ul>
+ *
+ * <p>General implementation:
+ *
+ * <ol>
+ * <li>Hash-index inserts by their non-PK field hash, for efficient Copy-on-Write detection.
+ * <li>Skip matching (delete, insert) pairs with identical non-PK columns. A CoW operation deletes
+ * and rewrites the whole file (minus some records that are actually marked for deletion).
+ * Unchanged records are no-ops and should not be mistaken for updates.
Review Comment:
"Unchanged records are no-ops"
Is this a quirk in Iceberg CDC ? Ideally we never produce such records.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergUtils.java:
##########
@@ -471,120 +473,150 @@ private static Object getIcebergTimestampValue(Object beamValue, boolean shouldA
}
}
+ /** Converts a {@link StructLike} to a Beam {@link Row}. */
+ public static Row structToRow(Schema schema, StructLike struct) {
+ checkState(
+ schema.getFieldCount() == struct.size(),
+ "Struct of size %s does not match expected schema size %s",
+ struct.size(),
+ schema.getFieldCount());
+ Row.Builder rowBuilder = Row.withSchema(schema);
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ Schema.Field field = schema.getField(i);
+ @Nullable Object icebergValue = struct.get(i, Object.class);
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
/** Converts an Iceberg {@link Record} to a Beam {@link Row}. */
public static Row icebergRecordToBeamRow(Schema schema, Record record) {
Row.Builder rowBuilder = Row.withSchema(schema);
for (Schema.Field field : schema.getFields()) {
- boolean isNullable = field.getType().getNullable();
@Nullable Object icebergValue = record.getField(field.getName());
- if (icebergValue == null) {
- if (isNullable) {
- rowBuilder.addValue(null);
- continue;
- }
- throw new RuntimeException(
- String.format("Received null value for required field '%s'.", field.getName()));
+ addIcebergValue(rowBuilder, field, icebergValue);
+ }
+ return rowBuilder.build();
+ }
+
+ private static void addIcebergValue(
+ Row.Builder rowBuilder, Schema.Field field, @Nullable Object icebergValue) {
+ boolean isNullable = field.getType().getNullable();
+ if (icebergValue == null) {
+ if (isNullable) {
+ rowBuilder.addValue(null);
+ return;
}
- switch (field.getType().getTypeName()) {
- case BYTE:
- case INT16:
- case INT32:
- case INT64:
- case DECIMAL: // Iceberg and Beam both use BigDecimal
- case FLOAT: // Iceberg and Beam both use float
- case DOUBLE: // Iceberg and Beam both use double
- case STRING: // Iceberg and Beam both use String
- case BOOLEAN: // Iceberg and Beam both use boolean
- rowBuilder.addValue(icebergValue);
- break;
- case ARRAY:
- checkState(
- icebergValue instanceof List,
- "Expected List type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
- Schema.FieldType collectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (collectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
- beamList =
- beamList.stream()
- .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
- .collect(Collectors.toList());
- }
- rowBuilder.addValue(beamList);
- break;
- case ITERABLE:
- checkState(
- icebergValue instanceof Iterable,
- "Expected Iterable type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
- Schema.FieldType iterableCollectionType =
- checkStateNotNull(field.getType().getCollectionElementType());
- // recurse on struct types
- if (iterableCollectionType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
- ImmutableList.Builder<Row> builder = ImmutableList.builder();
- for (Record v : (Iterable<@NonNull Record>) icebergValue) {
- builder.add(icebergRecordToBeamRow(innerSchema, v));
- }
- beamIterable = builder.build();
+ throw new RuntimeException(
+ String.format("Received null value for required field '%s'.", field.getName()));
+ }
+ switch (field.getType().getTypeName()) {
+ case BYTE:
+ case INT16:
+ case INT32:
+ case INT64:
+ case DECIMAL: // Iceberg and Beam both use BigDecimal
+ case FLOAT: // Iceberg and Beam both use float
+ case DOUBLE: // Iceberg and Beam both use double
+ case STRING: // Iceberg and Beam both use String
+ case BOOLEAN: // Iceberg and Beam both use boolean
+ rowBuilder.addValue(icebergValue);
+ break;
+ case ARRAY:
+ checkState(
+ icebergValue instanceof List,
+ "Expected List type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ List<@NonNull ?> beamList = (List<@NonNull ?>) icebergValue;
+ Schema.FieldType collectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (collectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(collectionType.getRowSchema());
+ beamList =
+ beamList.stream()
+ .map(v -> icebergRecordToBeamRow(innerSchema, (Record) v))
+ .collect(Collectors.toList());
+ }
+ rowBuilder.addValue(beamList);
+ break;
+ case ITERABLE:
+ checkState(
+ icebergValue instanceof Iterable,
+ "Expected Iterable type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Iterable<@NonNull ?> beamIterable = (Iterable<@NonNull ?>) icebergValue;
+ Schema.FieldType iterableCollectionType =
+ checkStateNotNull(field.getType().getCollectionElementType());
+ // recurse on struct types
+ if (iterableCollectionType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(iterableCollectionType.getRowSchema());
+ ImmutableList.Builder<Row> builder = ImmutableList.builder();
+ for (Record v : (Iterable<@NonNull Record>) icebergValue) {
+ builder.add(icebergRecordToBeamRow(innerSchema, v));
}
- rowBuilder.addValue(beamIterable);
- break;
- case MAP:
- checkState(
- icebergValue instanceof Map,
- "Expected Map type for field '%s' but received %s",
- field.getName(),
- icebergValue.getClass());
- Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
- Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
- // recurse on struct types
- if (valueType.getTypeName().isCompositeType()) {
- Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
- ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
- for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
- Record rec = ((Record) entry.getValue());
- newMap.put(
- checkStateNotNull(entry.getKey()),
- icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
- }
- beamMap = newMap.build();
+ beamIterable = builder.build();
+ }
+ rowBuilder.addValue(beamIterable);
+ break;
+ case MAP:
+ checkState(
+ icebergValue instanceof Map,
+ "Expected Map type for field '%s' but received %s",
+ field.getName(),
+ icebergValue.getClass());
+ Map<?, ?> beamMap = (Map<?, ?>) icebergValue;
+ Schema.FieldType valueType = checkStateNotNull(field.getType().getMapValueType());
+ // recurse on struct types
+ if (valueType.getTypeName().isCompositeType()) {
+ Schema innerSchema = checkStateNotNull(valueType.getRowSchema());
+ ImmutableMap.Builder<Object, Row> newMap = ImmutableMap.builder();
+ for (Map.Entry<?, ?> entry : ((Map<?, ?>) icebergValue).entrySet()) {
+ Record rec = ((Record) entry.getValue());
+ newMap.put(
+ checkStateNotNull(entry.getKey()),
+ icebergRecordToBeamRow(innerSchema, checkStateNotNull(rec)));
}
- rowBuilder.addValue(beamMap);
- break;
- case DATETIME:
- // Iceberg uses a long for micros.
- // Beam DATETIME uses joda's DateTime, which only supports millis,
- // so we do lose some precision here
- rowBuilder.addValue(getBeamDateTimeValue(icebergValue));
- break;
- case BYTES:
- // Iceberg uses ByteBuffer; Beam uses byte[]
- rowBuilder.addValue(((ByteBuffer) icebergValue).array());
- break;
- case ROW:
- Record nestedRecord = (Record) icebergValue;
- Schema nestedSchema =
- checkArgumentNotNull(
- field.getType().getRowSchema(),
- "Corrupted schema: Row type did not have associated nested schema.");
- rowBuilder.addValue(icebergRecordToBeamRow(nestedSchema, nestedRecord));
- break;
- case LOGICAL_TYPE:
- rowBuilder.addValue(getLogicalTypeValue(icebergValue, field.getType()));
- break;
- default:
+ beamMap = newMap.build();
+ }
+ rowBuilder.addValue(beamMap);
+ break;
+ case DATETIME:
+ // Iceberg uses a long for micros.
+ // Beam DATETIME uses joda's DateTime, which only supports millis,
+ // so we do lose some precision here
Review Comment:
Needs a TODO to fix ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/OverlapRange.java:
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.util.Comparator;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.values.Row;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Primary-key-projection and overlap-range comparison helper.
+ *
+ * <p>Used by {@link LocalResolveDoFn} and {@link ReadFromChangelogs} to decide whether a record's
Review Comment:
In general, ranges where both sides are inclusive can lead to errors at the boundary. For example, this is why our source split ranges are non-inclusive at the upper bound "[...)". I'm not sure if it's an issue here but if we keep range as is, let's add assertions/tests to make sure that we do not run into issues at the boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3589979900
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
Review Comment:
I added a scaling factor in ChangelogScanner to make a rough estimate of the decoded bytes. This reduces the byte size threshold of batches getting routed to LocalResolveDoFn, so we won't see big batches making it there. Should help us avoid OOMs, and we can adjust the factor when we see how it's handled with real usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3565677045
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/LocalResolveDoFn.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.apache.iceberg.util.StructLikeUtil;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Resolves a small bi-directional changelog group entirely in memory. This is the equivalent of
+ * {@link ReadFromChangelogs} + {@link CoGroupByKey} + {@link ResolveChanges}.
+ *
+ * <p>All tasks in a changelog group belong to the same Iceberg {@link Snapshot}. The upstream
+ * {@link ChangelogScanner} routes here only when the total size of the bi-directional group fits
+ * within {@link TableProperties#SPLIT_SIZE}.
+ *
+ * <p>The incoming batch's overlap region has already been computed in the scanning phase by {@link
+ * ChangelogScanner}. In this DoFn, we just process each task and route records:
+ *
+ * <ul>
+ * <li>Records whose PK falls <b>outside</b> the overlap range cannot have an opposing-side match,
+ * so they are emitted directly with {@code INSERT} or {@code DELETE} kind.
+ * <li>Records whose PK falls <b>inside</b> the overlap range are stashed in a {@link
+ * StructLikeMap} keyed by PK, then resolved by {@link CdcResolver}.
+ * </ul>
+ */
+class LocalResolveDoFn extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, Row> {
+ private final IcebergScanConfig scanConfig;
+ private final org.apache.beam.sdk.schemas.Schema projectedBeamSchema;
+ private final org.apache.beam.sdk.schemas.Schema outputBeamSchema;
+
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull List<Types.NestedField> nonPkFields;
+ private transient @MonotonicNonNull StructProjection projector;
+
+ LocalResolveDoFn(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ this.projectedBeamSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ }
+
+ @Setup
+ public void setup() {
+ Schema tableSchema =
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier()).schema();
+ Schema fullReadSchema =
+ CdcOutputUtils.readSchemaWithRowMetadata(scanConfig.getMetadataColumns(), tableSchema);
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ Set<String> pkFieldNames = new HashSet<>(overlap.recordIdSchema().identifierFieldNames());
+ // The dedup logic only inspects non-PK fields, so precompute them once.
+ List<Types.NestedField> nonPk = new ArrayList<>();
+ for (Types.NestedField f : tableSchema.columns()) {
+ if (!pkFieldNames.contains(f.name())) {
+ nonPk.add(f);
+ }
+ }
+ this.nonPkFields = nonPk;
+ this.projector =
+ StructProjection.create(
+ fullReadSchema,
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ OutputReceiver<Row> out)
+ throws IOException {
+ ChangelogDescriptor descriptor = element.getKey();
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+ OverlapRange ovl = checkStateNotNull(overlap);
+
+ // {PK: (inserts | deletes)} for in-overlap records that need resolution.
+ // Records outside the overlap are emitted directly
+ StructLikeMap<PkGroup> pkGroups = StructLikeMap.create(ovl.recordIdSchema().asStruct());
+
+ @Nullable StructLike overlapLower = ovl.toStructLike(descriptor.getOverlapLower());
+ @Nullable StructLike overlapUpper = ovl.toStructLike(descriptor.getOverlapUpper());
+ for (SerializableChangelogTask task : element.getValue()) {
+ readAndRoute(descriptor, task, table, overlapLower, overlapUpper, pkGroups, out);
+ }
+
+ resolveAndEmit(descriptor, pkGroups, table.schema(), out);
Review Comment:
done
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,493 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
+ outputReceiver
+ .get(taggedOutput)
+ .builder(
+ KV.of(
+ CdcRowDescriptor.builder()
+ .setCommitSnapshotId(commitSnapshotId)
+ .setSnapshotSequenceNumber(commitSnapshotSequenceNumber)
+ .setPrimaryKey(pk)
+ .build(),
+ row))
+ .setValueKind(kind)
+ .output();
+
+ } else {
+ // outside overlap -- get projected record and output
+ StructLike projected = outputProjector().wrap(rec);
+ Row row = structToRow(projectedBeamRowSchema, projected);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ }
+ }
+ }
+
+ trackMetrics(task.getType());
+ }
+
+ private StructProjection outputProjector() {
+ if (outputProjector == null) {
+ outputProjector =
+ StructProjection.create(
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(),
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema()),
+ CdcOutputUtils.readSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema()));
+ }
+ return outputProjector;
+ }
+
+ private StructProjection pkProjector() {
+ if (pkProjector == null) {
+ pkProjector =
+ StructProjection.create(
+ TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier())
+ .schema(),
+ scanConfig.recordIdSchema());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3590010866
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/ReadFromChangelogs.java:
##########
@@ -0,0 +1,495 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergRecordToBeamRow;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.icebergSchemaToBeamSchema;
+import static org.apache.beam.sdk.io.iceberg.IcebergUtils.structToRow;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.LARGE_BIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.ChangelogScanner.UNIDIRECTIONAL_TASKS;
+import static org.apache.beam.sdk.io.iceberg.cdc.SerializableChangelogTask.Type.ADDED_ROWS;
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.io.iceberg.IcebergScanConfig;
+import org.apache.beam.sdk.io.iceberg.IcebergUtils;
+import org.apache.beam.sdk.io.iceberg.SerializableDeleteFile;
+import org.apache.beam.sdk.io.iceberg.TableCache;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PInput;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.ChangelogScanTask;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.util.StructProjection;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * A {@link PTransform} that processes batches of {@link ChangelogScanTask}s and routes them
+ * accordingly:
+ *
+ * <ul>
+ * <li>Records from Uni-directional batches are directly emitted, as INSERT or DELETE kind
+ * <li>Records from Bi-directional batches are compared against the Primary Key overlap range:
+ * <ul>
+ * <li>if outside the overlap, emit directly as INSERT or DELETE kind
+ * <li>if inside the overlap, key by (snapshot seq#, pk) and route to downstream {@link
+ * CoGroupByKey} and final resolution by {@link ResolveChanges}
+ * </ul>
+ * </ul>
+ *
+ * <p>We first key bi-directional rows by (snapshot sequence number, primary key) before sending to
+ * {@link CoGroupByKey} to ensure they stay isolated from other PKs or snapshots. Inserts are routed
+ * to
+ *
+ * <p>A {@link ChangelogScanTask} comes in three types:
+ *
+ * <ol>
+ * <li><b>AddedRowsScanTask</b>: Indicates records have been inserted by a new DataFile.
+ * <li><b>DeletedRowsScanTask</b>: Indicates records have been deleted using a DeleteFile.
+ * <li><b>DeletedDataFileScanTask</b>: Indicates a whole DataFile has been deleted.
+ * </ol>
+ *
+ * <p>Each of these types need to be processed differently. More details in {@link
+ * CdcReadUtils#changelogRecordsForTask}.
+ *
+ * <p>CDC metadata has two entry points in this transform. Row metadata columns are requested from
+ * the Iceberg reader by {@link CdcReadUtils} and travel inside intermediate rows until final output
+ * assembly. Snapshot metadata columns come from the {@link ChangelogDescriptor} / {@link
+ * CdcRowDescriptor} carried with each task or shuffled row, and {@code _change_type} comes from the
+ * emitted change kind. Final user-visible rows are assembled by {@link CdcOutputUtils#outputRow},
+ * which appends all requested metadata as top-level columns in the configured order.
+ */
+public class ReadFromChangelogs extends PTransform<PCollectionTuple, ReadFromChangelogs.Output> {
+ private static final Counter numAddedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numAddedRowsScanTasksCompleted");
+ private static final Counter numDeletedRowsScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedRowsScanTasksCompleted");
+ private static final Counter numDeletedDataFileScanTasksCompleted =
+ Metrics.counter(ReadFromChangelogs.class, "numDeletedDataFileScanTasksCompleted");
+
+ private static final TupleTag<Row> UNIDIRECTIONAL_ROWS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_INSERTS = new TupleTag<>();
+ private static final TupleTag<KV<CdcRowDescriptor, Row>> BIDIRECTIONAL_DELETES = new TupleTag<>();
+
+ private final IcebergScanConfig scanConfig;
+
+ ReadFromChangelogs(IcebergScanConfig scanConfig) {
+ this.scanConfig = scanConfig;
+ }
+
+ @Override
+ public org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output expand(
+ PCollectionTuple input) {
+ Schema fullRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ Schema projectedRowSchema =
+ IcebergUtils.icebergSchemaToBeamSchema(scanConfig.getProjectedSchema());
+ Schema outputRowSchema = CdcOutputUtils.outputSchema(scanConfig, projectedRowSchema);
+
+ // === UNIDIRECTIONAL tasks ===
+ // (i.e. only deletes, or only inserts)
+ // take the fast approach of just reading and emitting CDC records
+ PCollection<Row> uniDirectionalRows =
+ input
+ .get(UNIDIRECTIONAL_TASKS)
+ .apply("Redistribute Uni-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Uni-Directional Changes",
+ ParDo.of(ReadDoFn.unidirectional(scanConfig))
+ .withOutputTags(UNIDIRECTIONAL_ROWS, TupleTagList.empty()))
+ .get(UNIDIRECTIONAL_ROWS)
+ .setRowSchema(outputRowSchema);
+
+ // === BIDIRECTIONAL tasks ===
+ // (i.e. a task group containing a mix of deletes and inserts)
+ // read and route records according to their PK (see class java doc)
+ PCollectionTuple biDirectionalRows =
+ input
+ .get(LARGE_BIDIRECTIONAL_TASKS)
+ .apply("Redistribute Large Bi-Directional Changes", Redistribute.arbitrarily())
+ .apply(
+ "Read Bi-Directional Changes",
+ ParDo.of(ReadDoFn.bidirectional(scanConfig))
+ .withOutputTags(
+ BIDIRECTIONAL_INSERTS,
+ TupleTagList.of(BIDIRECTIONAL_DELETES).and(UNIDIRECTIONAL_ROWS)));
+ // Collect pruned (non-overlapping) rows from bi-directional reader
+ PCollection<Row> nonOverlappingRowsFromBiDirTasks =
+ biDirectionalRows.get(UNIDIRECTIONAL_ROWS).setRowSchema(outputRowSchema);
+
+ // Flatten uni-directional rows from both sources
+ PCollection<Row> allUniDirectionalRows =
+ PCollectionList.of(uniDirectionalRows)
+ .and(nonOverlappingRowsFromBiDirTasks)
+ .apply("Flatten Uni-Directional Rows", Flatten.pCollections());
+
+ // Reify to preserve each record's timestamp (CoGBK overwrites timestamps with the window's
+ // end-of-window)
+ // Note: element timestamps are snapshot commit timestamp
+ KvCoder<CdcRowDescriptor, Row> keyedOutputCoder =
+ KvCoder.of(
+ CdcRowDescriptor.coder(scanConfig.rowIdBeamSchema()), SchemaCoder.of(fullRowSchema));
+ PCollection<KV<CdcRowDescriptor, Row>> keyedInsertsWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_INSERTS).setCoder(keyedOutputCoder);
+ PCollection<KV<CdcRowDescriptor, Row>> keyedDeletesWithTimestamps =
+ biDirectionalRows.get(BIDIRECTIONAL_DELETES).setCoder(keyedOutputCoder);
+
+ return new org.apache.beam.sdk.io.iceberg.cdc.ReadFromChangelogs.Output(
+ input.getPipeline(),
+ allUniDirectionalRows,
+ keyedInsertsWithTimestamps,
+ keyedDeletesWithTimestamps);
+ }
+
+ public static class Output implements POutput {
+ private final Pipeline pipeline;
+ private final PCollection<Row> uniDirectionalRows;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts;
+ private final PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes;
+
+ Output(
+ Pipeline p,
+ PCollection<Row> uniDirectionalRows,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts,
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes) {
+ this.pipeline = p;
+ this.uniDirectionalRows = uniDirectionalRows;
+ this.biDirectionalInserts = biDirectionalInserts;
+ this.biDirectionalDeletes = biDirectionalDeletes;
+ }
+
+ PCollection<Row> uniDirectionalRows() {
+ return uniDirectionalRows;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalInserts() {
+ return biDirectionalInserts;
+ }
+
+ PCollection<KV<CdcRowDescriptor, Row>> biDirectionalDeletes() {
+ return biDirectionalDeletes;
+ }
+
+ @Override
+ public Pipeline getPipeline() {
+ return pipeline;
+ }
+
+ @Override
+ public Map<TupleTag<?>, PValue> expand() {
+ return ImmutableMap.of(
+ UNIDIRECTIONAL_ROWS,
+ uniDirectionalRows,
+ BIDIRECTIONAL_INSERTS,
+ biDirectionalInserts,
+ BIDIRECTIONAL_DELETES,
+ biDirectionalDeletes);
+ }
+
+ @Override
+ public void finishSpecifyingOutput(
+ String transformName, PInput input, PTransform<?, ?> transform) {}
+ }
+
+ @DoFn.BoundedPerElement
+ private static class ReadDoFn<OutT>
+ extends DoFn<KV<ChangelogDescriptor, List<SerializableChangelogTask>>, OutT> {
+ private final IcebergScanConfig scanConfig;
+ private final boolean keyedOutput;
+ private final Schema projectedBeamRowSchema;
+ private final Schema outputBeamRowSchema;
+ private final Schema fullBeamRowSchema;
+ private transient @MonotonicNonNull OverlapRange overlap;
+ private transient @MonotonicNonNull StructProjection outputProjector;
+ private transient @MonotonicNonNull StructProjection pkProjector;
+
+ /** Used for uni-directional changes. Records are output immediately as-is. */
+ static ReadDoFn<Row> unidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, false);
+ }
+
+ /**
+ * Used for bi-directional changes. Records are keyed by (snapshot sequence number, primary key)
+ * and sent to a CoGBK.
+ */
+ static ReadDoFn<KV<CdcRowDescriptor, Row>> bidirectional(IcebergScanConfig scanConfig) {
+ return new ReadDoFn<>(scanConfig, true);
+ }
+
+ private ReadDoFn(IcebergScanConfig scanConfig, boolean keyedOutput) {
+ this.scanConfig = scanConfig;
+ this.keyedOutput = keyedOutput;
+
+ this.projectedBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getProjectedSchema());
+ this.outputBeamRowSchema =
+ CdcOutputUtils.outputSchema(
+ scanConfig, icebergSchemaToBeamSchema(scanConfig.getProjectedSchema()));
+ this.fullBeamRowSchema =
+ CdcOutputUtils.readBeamSchemaWithRowMetadata(
+ scanConfig.getMetadataColumns(), scanConfig.getSchema());
+ }
+
+ @Setup
+ public void setup() {
+ this.overlap = OverlapRange.forScanConfig(scanConfig);
+ }
+
+ @ProcessElement
+ public void process(
+ @Element KV<ChangelogDescriptor, List<SerializableChangelogTask>> element,
+ RestrictionTracker<OffsetRange, Long> tracker,
+ MultiOutputReceiver out)
+ throws IOException {
+ Table table = TableCache.get(scanConfig.getCatalogConfig(), scanConfig.getTableIdentifier());
+
+ List<SerializableChangelogTask> tasks = element.getValue();
+ ChangelogDescriptor descriptor = element.getKey();
+ @Nullable Row overlapLower = descriptor.getOverlapLower();
+ @Nullable Row overlapUpper = descriptor.getOverlapUpper();
+
+ for (long l = tracker.currentRestriction().getFrom();
+ l < tracker.currentRestriction().getTo();
+ l++) {
+ if (!tracker.tryClaim(l)) {
+ return;
+ }
+
+ SerializableChangelogTask task = tasks.get((int) l);
+ processTaskRecords(descriptor, task, overlapLower, overlapUpper, table, out);
+ }
+ }
+
+ /**
+ * Processes a ChangelogScanTask and routes records accordingly:
+ *
+ * <p>If this DoFn is configured with {@link #unidirectional}, we simply read records and output
+ * directly to {@link #UNIDIRECTIONAL_ROWS}.
+ *
+ * <p>If this DoFn is configured with {@link #bidirectional}, we compare against the Primary Key
+ * overlap range. If within the overlap, we key by (snapshotId, PK) and out to either {@link
+ * #BIDIRECTIONAL_INSERTS} or {@link #BIDIRECTIONAL_DELETES}. Otherwise (not in overlap), we
+ * output the record directly to {@link #UNIDIRECTIONAL_ROWS}.
+ */
+ private void processTaskRecords(
+ ChangelogDescriptor descriptor,
+ SerializableChangelogTask task,
+ @Nullable Row overlapLowerRow,
+ @Nullable Row overlapUpperRow,
+ Table table,
+ MultiOutputReceiver outputReceiver)
+ throws IOException {
+ OverlapRange ovl = checkStateNotNull(overlap);
+ @Nullable StructLike overlapLower = ovl.toStructLike(overlapLowerRow);
+ @Nullable StructLike overlapUpper = ovl.toStructLike(overlapUpperRow);
+
+ boolean isInsert = task.getType() == ADDED_ROWS;
+ TupleTag<KV<CdcRowDescriptor, Row>> taggedOutput =
+ isInsert ? BIDIRECTIONAL_INSERTS : BIDIRECTIONAL_DELETES;
+ ValueKind kind = isInsert ? ValueKind.INSERT : ValueKind.DELETE;
+ long commitSnapshotId = descriptor.getCommitSnapshotId();
+ long commitSnapshotSequenceNumber = descriptor.getSnapshotSequenceNumber();
+
+ Schema readSchema = keyedOutput ? fullBeamRowSchema : projectedBeamRowSchema;
+ try (CloseableIterable<Record> records =
+ CdcReadUtils.changelogRecordsForTask(task, table, scanConfig, !keyedOutput)) {
+ for (Record rec : records) {
+ // uni-directional -- just output records (they are already projected by read pushdown)
+ if (!keyedOutput) {
+ Row row = icebergRecordToBeamRow(projectedBeamRowSchema, rec);
+ outputReceiver
+ .get(UNIDIRECTIONAL_ROWS)
+ .builder(
+ CdcOutputUtils.outputRow(
+ scanConfig.getMetadataColumns(),
+ outputBeamRowSchema,
+ descriptor,
+ kind,
+ row))
+ .setValueKind(kind)
+ .output();
+ continue;
+ }
+
+ // bi-directional -- compare overlap
+ if (ovl.contains(rec, overlapLower, overlapUpper)) {
+ // inside overlap -- read full row and output KV
+ Row row = icebergRecordToBeamRow(readSchema, rec);
+ Row pk = structToRow(scanConfig.rowIdBeamSchema(), pkProjector().wrap(rec));
Review Comment:
Will wait until that PR gets merged to merge this one. But we're good to go for another review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38837:
URL: https://github.com/apache/beam/pull/38837#issuecomment-4949989361
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #38837:
URL: https://github.com/apache/beam/pull/38837#discussion_r3707829565
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcResolver.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg.cdc;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.ValueKind;
+import org.apache.iceberg.data.Record;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * Helper class to reconcile CDC rows. Used by {@link ResolveChanges} (with Beam {@link Row}s) and
+ * {@link LocalResolveDoFn} (with Iceberg {@link Record}s).
+ *
+ * <p>We determine the output ValueKind as follows:
+ *
+ * <ul>
+ * <li>(delete, insert) pairs become {@code UPDATE_BEFORE} + {@code UPDATE_AFTER}
Review Comment:
Yep, added a some language above to clarify
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java:
##########
@@ -70,6 +70,7 @@
*/
public final class CdcReadUtils {
private static final Logger LOG = LoggerFactory.getLogger(CdcReadUtils.class);
+ static final int COMPRESSED_TO_DECODED_BYTES_ESTIMATE = 4;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Iceberg CDC] Add Changelog readers and update resolver [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #38837:
URL: https://github.com/apache/beam/pull/38837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org