You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2026/10/12 21:40:29 UTC
[PR] Add YAML tests for writing to Iceberg CDC sink [beam]
ahmedabu98 opened a new pull request, #40542:
URL: https://github.com/apache/beam/pull/40542
Adds YAML tests for:
1. Delta to Iceberg CDC (batch)
2. Iceberg to Iceberg CDC (streaming and batch)
Adds a small fix to let Python recognize columns that start with an underscore `"-"` when converting Beam Rows to native objects (needed for xlang, yaml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #40542:
URL: https://github.com/apache/beam/pull/40542#issuecomment-6148289142
4 reviewers is too many :) assigning to Cham since you have the best context and have already started the review - thanks
R: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #40542:
URL: https://github.com/apache/beam/pull/40542#discussion_r4265495182
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/sink/OrderedCommitFn.java:
##########
@@ -157,7 +157,6 @@ interface Clock extends LongSupplier, Serializable {}
this.pendingSpec = StateSpecs.bag(CommitDeltas.windowedCommitCoder());
}
- @RequiresStableInput
Review Comment:
Done. It would only change behavior for Flink, but we need it to run Python/YAML pipelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40542:
URL: https://github.com/apache/beam/pull/40542#issuecomment-6148315334
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #40542:
URL: https://github.com/apache/beam/pull/40542#discussion_r4265517318
##########
sdks/python/apache_beam/yaml/extended_tests/e2e/delta_lake_cdc_to_iceberg.yaml:
##########
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
Review Comment:
We don't seem to have Dataflow YAML tests set up. I ran the following on Dataflow though and got the expected output:
DeltaLake to Iceberg:
```yaml
pipeline:
type: chain
transforms:
- type: ReadFromDeltaCDC
config:
table: "gs://ahmedabualsaud/delta/orders"
start_version: 0
end_version: 3
include_metadata_columns: ["_change_type", "_commit_version"]
hadoop_config:
"fs.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
"fs.AbstractFileSystem.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
"fs.gs.auth.type": "APPLICATION_DEFAULT"
"fs.gs.project.id": "apache-beam-testing"
- type: WriteToIceberg
config:
table: "shop.orders_from_delta"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
mode: "merge-on-read"
equality_columns: ["id"]
change_type_column: "_change_type"
change_type_map:
insert: "INSERT"
update_preimage: "UPDATE_BEFORE"
update_postimage: "UPDATE_AFTER"
delete: "DELETE"
sequence_number_column: "_commit_version"
```
Iceberg to Iceberg:
```yaml
pipeline:
type: chain
transforms:
- type: ReadFromIcebergCDC
config:
table: "shop.orders_from_delta"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
config_properties:
"fs.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
"fs.AbstractFileSystem.gs.impl": "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
"fs.gs.auth.type": "APPLICATION_DEFAULT"
"fs.gs.project.id": "apache-beam-testing"
include_metadata_columns: ["_change_type", "_commit_snapshot_sequence_number"]
- type: WriteToIceberg
config:
table: "shop.orders_from_iceberg"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
mode: "merge-on-read"
equality_columns: ["id"]
change_type_column: "_change_type"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #40542:
URL: https://github.com/apache/beam/pull/40542#discussion_r4253487268
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/sink/OrderedCommitFn.java:
##########
@@ -157,7 +157,6 @@ interface Clock extends LongSupplier, Serializable {}
this.pendingSpec = StateSpecs.bag(CommitDeltas.windowedCommitCoder());
}
- @RequiresStableInput
Review Comment:
Could you mention in the PR description why these were removed ? Won't this change the behavior for non-Dataflow runners ?
##########
sdks/python/apache_beam/yaml/extended_tests/e2e/delta_lake_cdc_to_iceberg.yaml:
##########
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
Review Comment:
Could you also try these against Dataflow ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40542:
URL: https://github.com/apache/beam/pull/40542#issuecomment-6134000524
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40542?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `21.42857%` with `11 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 58.65%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d2063be`](https://app.codecov.io/gh/apache/beam/commit/d2063be364557fd7080f5a7c56440be2fbd46e80?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 139 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/40542?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [sdks/python/apache\_beam/yaml/integration\_tests.py](https://app.codecov.io/gh/apache/beam/pull/40542?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Fyaml%2Fintegration_tests.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0veWFtbC9pbnRlZ3JhdGlvbl90ZXN0cy5weQ==) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/40542?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #40542 +/- ##
============================================
+ Coverage 56.16% 58.65% +2.48%
- Complexity 2288 13949 +11661
============================================
Files 1124 2593 +1469
Lines 178336 272252 +93916
Branches 1489 11286 +9797
============================================
+ Hits 100170 159697 +59527
- Misses 75645 106534 +30889
- Partials 2521 6021 +3500
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/40542/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [java](https://app.codecov.io/gh/apache/beam/pull/40542/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.03% <ø> (-10.75%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40542?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #40542:
URL: https://github.com/apache/beam/pull/40542#issuecomment-6135953029
Thanks! Added few more comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40542:
URL: https://github.com/apache/beam/pull/40542#issuecomment-6134009794
Assigning reviewers:
R: @tvalentyn for label python.
R: @damccorm for label java.
R: @derrickaw for label yaml.
R: @chamikaramj for label iceberg.
This pull request likely touches a core component ("core" label). Please review with scrutiny.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add YAML tests for writing to Iceberg CDC sink [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #40542:
URL: https://github.com/apache/beam/pull/40542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org