You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tkaymak (via GitHub)" <gi...@apache.org> on 2025/04/28 14:58:30 UTC

[PR] Update Debezium in DebeziumIO to 3.1.1 [beam]

tkaymak opened a new pull request, #34763:
URL: https://github.com/apache/beam/pull/34763

   This is step 2 for issue https://github.com/apache/beam/issues/34747 to get to a newer Debezium (3.1.1) version in Beam, as Debezium requires Java 17 or later, this MUST NOT be merged before #34751. I tested the local build (+ all tests) with gradle and Java 17 before making this change.
   
   This build and its tests will fail without the PR #34751 being merged first.
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [X] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [X] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2096107223


##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOPostgresSqlConnectorIT.java:
##########
@@ -173,7 +175,6 @@ public void testDebeziumIOPostgresSql() {
                         .withPort(port)
                         .withConnectionProperty("database.dbname", "inventory")
                         .withConnectionProperty("database.server.name", "dbserver1")
-                        .withConnectionProperty("database.include.list", "inventory")

Review Comment:
   Seems necessary as per https://stackoverflow.com/questions/72174047/debezium-postgres-connector-after-applying-the-include-exclude-list-filters-no



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2138766311


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -90,13 +135,8 @@ task integrationTest(type: Test, dependsOn: processTestResources) {
   }
 }
 
-configurations.all (Configuration it) -> {
-  resolutionStrategy {
-    // Force protobuf 3 because debezium is currently incompatible with protobuf 4.
-    // TODO - remove this and upgrade the version of debezium once a proto-4 compatible version is available
-    // https://github.com/apache/beam/pull/33526 does some of this, but was abandoned because it still doesn't
-    // work with protobuf 4.
-    force "com.google.protobuf:protobuf-java:3.25.5"
-    force "com.google.protobuf:protobuf-java-util:3.25.5"
-  }
+project.tasks.each {

Review Comment:
   I mean also remove these project.tasks.each { ... } (as well as in expansion-service build.gradle). It will lead to DebeziumIO get skipped when building a release without #35232
   
   This has already been handled in https://github.com/apache/beam/blob/de76cfa16d36e2cb8dd1a1092710f0cd477e0f8d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1856



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2891864202

   @derrickaw not ready to review yet, finally I got the failing test outputs I needed to continue working. I will request a review once it's ready!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2138786388


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -38,40 +40,83 @@ dependencies {
     implementation library.java.joda_time
     provided library.java.jackson_dataformat_csv
     permitUnusedDeclared library.java.jackson_dataformat_csv
-    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
-    testImplementation project(path: ":sdks:java:io:common")
+
+    // Kafka connect dependencies
+    implementation "org.apache.kafka:connect-api:3.9.0"
+
+    // Debezium dependencies
+    implementation group: 'io.debezium', name: 'debezium-core', version: '3.1.1.Final'
 
     // Test dependencies
+    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
+    testImplementation project(path: ":sdks:java:io:common")
     testImplementation library.java.junit
     testImplementation project(path: ":sdks:java:io:jdbc")
     testRuntimeOnly library.java.slf4j_jdk14
     testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
     testImplementation project(":runners:google-cloud-dataflow-java")
     testImplementation library.java.hamcrest
     testImplementation library.java.testcontainers_base
-    testImplementation library.java.testcontainers_mysql
-    testImplementation library.java.testcontainers_postgresql
-    // TODO(https://github.com/apache/beam/issues/31678) HikariCP 5.x requires Java11+
-    testImplementation 'com.zaxxer:HikariCP:4.0.3'
+    testImplementation "org.testcontainers:kafka"
+    testImplementation "org.testcontainers:mysql"
+    testImplementation "org.testcontainers:postgresql"
+    testImplementation "io.debezium:debezium-testing-testcontainers:3.1.1.Final"
+    testImplementation 'com.zaxxer:HikariCP:5.1.0'
 
-    // Kafka connect dependencies
-    implementation "org.apache.kafka:connect-api:2.5.0"
-    implementation "org.apache.kafka:connect-json:2.5.0"
-    permitUnusedDeclared "org.apache.kafka:connect-json:2.5.0" // BEAM-11761
+    // Debezium connector implementations for testing
+    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '3.1.1.Final'
+    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '3.1.1.Final'
+}
 
-    // Debezium dependencies
-    implementation group: 'io.debezium', name: 'debezium-core', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '1.3.1.Final'
+// TODO: Remove pin after Beam has unpinned it

Review Comment:
   Thanks, opened go/beampr/35231



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2935357641

   @Abacn - it's not ready for review yet, but tests are passing now. 
   I plan to refactor and make it review-ready in the coming days. Thank you for your patience 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2138766311


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -90,13 +135,8 @@ task integrationTest(type: Test, dependsOn: processTestResources) {
   }
 }
 
-configurations.all (Configuration it) -> {
-  resolutionStrategy {
-    // Force protobuf 3 because debezium is currently incompatible with protobuf 4.
-    // TODO - remove this and upgrade the version of debezium once a proto-4 compatible version is available
-    // https://github.com/apache/beam/pull/33526 does some of this, but was abandoned because it still doesn't
-    // work with protobuf 4.
-    force "com.google.protobuf:protobuf-java:3.25.5"
-    force "com.google.protobuf:protobuf-java-util:3.25.5"
-  }
+project.tasks.each {

Review Comment:
   I mean also remove these project.tasks.each { ... } (as well as in expansion-service build.gradle)
   
   This has already been handled in https://github.com/apache/beam/blob/de76cfa16d36e2cb8dd1a1092710f0cd477e0f8d/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1856



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2972065431

   #35232 is merged. Merging this PR for now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2069327676


##########
.gitignore:
##########
@@ -34,6 +34,9 @@ sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources
 sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/src/
 sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/sample.txt
 
+# Ignore generated debezium data
+sdks/java/io/debezium/data/

Review Comment:
   I do agree
   That is originating from the original PR here -> https://github.com/apache/beam/pull/33526/files#diff-bc37d034bad564583790a46f19d807abfe519c5671395fd494d8cce506c42947R37
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2965971880

   > > Thanks!
   > > We'd need to make some adjustments to CI to get DebeziumIO build and tested for cross-language tests. Let me do a separate PR, then will merge when infra is ready
   > 
   > Need some adjustment to make PostCommit Python working. Simply bump java version would fail Spark runner tests that do not support Java 21 (yet) https://github.com/apache/beam/actions/runs/15589069843/job/43903046792
   
   Thank you! Let me know if I can help you with that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137676267


##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOMySqlConnectorIT.java:
##########
@@ -98,19 +122,22 @@ private void monitorEssentialMetrics() {
           rs.close();
           Thread.sleep(4000);
         } else {
-          throw new IllegalArgumentException("OIOI");
+          throw new IllegalArgumentException(
+              "Illegal Argument Exception in monitorEssentialMetrics.");
         }
       }
-    } catch (InterruptedException | SQLException ex) {
-      throw new IllegalArgumentException("Oi", ex);
+    } catch (SQLException ex) {
+      LOG.error("SQL error in monitoring thread. Shutting down.", ex);

Review Comment:
   Was not intended, thank you for the catch, it's thrown now and causing the test to fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137676267


##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOMySqlConnectorIT.java:
##########
@@ -98,19 +122,22 @@ private void monitorEssentialMetrics() {
           rs.close();
           Thread.sleep(4000);
         } else {
-          throw new IllegalArgumentException("OIOI");
+          throw new IllegalArgumentException(
+              "Illegal Argument Exception in monitorEssentialMetrics.");
         }
       }
-    } catch (InterruptedException | SQLException ex) {
-      throw new IllegalArgumentException("Oi", ex);
+    } catch (SQLException ex) {
+      LOG.error("SQL error in monitoring thread. Shutting down.", ex);

Review Comment:
   Was not intended, thank you for the catch! It's thrown now and causing the test to fail (throw is in the next line).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2959694172

   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2962874822

   I got confirmation that this PR and update of Debezium to v3 also fixes the AlloyDB path of [AlloyDB -> CDC via Debezium in a Dataflow pipeline], which was one of the reasons why I picked up this work as it was apparently not working with DebeziumIO v1. 
   Do you think it makes sense to mention this somewhere?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2946382597

   Assigning reviewers:
   
   R: @robertwb for label java.
   R: @Abacn for label build.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2140560532


##########
sdks/java/io/debezium/src/README.md:
##########
@@ -164,9 +164,9 @@ By default, DebeziumIO initializes it with the former, though user may choose th
 
 ### Requirements and Supported versions
 
--  JDK v8
--  Debezium Connectors v1.3
--  Apache Beam 2.25
+-  JDK v17
+-  Debezium Connectors v3.1
+-  Apache Beam 2.65

Review Comment:
   ```suggestion
   -  Apache Beam 2.66
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2136710448


##########
CHANGES.md:
##########
@@ -184,6 +184,7 @@ N/A
 * [Python] Reshuffle now correctly respects user-specified type hints, fixing a previous bug where it might use FastPrimitivesCoder wrongly. This change could break pipelines with incorrect type hints in Reshuffle. If you have issues after upgrading, temporarily set update_compatibility_version to a previous Beam version to use the old behavior. The recommended solution is to fix the type hints in your code. ([#33932](https://github.com/apache/beam/pull/33932))
 * [Java] SparkReceiver 2 has been moved to SparkReceiver 3 that supports Spark 3.x. ([#33574](https://github.com/apache/beam/pull/33574))
 * [Python] Correct parsing of `collections.abc.Sequence` type hints was added, which can lead to pipelines failing type hint checks that were previously passing erroneously. These issues will be most commonly seen trying to consume a PCollection with a `Sequence` type hint after a GroupByKey or a CoGroupByKey. ([#33999](https://github.com/apache/beam/pull/33999)).
+* Debezium IO (Java) has been upgraded from depending on version 1.3.1.Final of io.debezium to 3.1.1.Final. This may cause some breaking changes since the libraries do not maintain full compatibility ([#33526](https://github.com/apache/beam/issues/33526)).

Review Comment:
   Remove this. It is 2.64.0 Section here



##########
CHANGES.md:
##########
@@ -31,7 +31,7 @@
 
 ## I/Os
 
-* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Debezium IO upgraded to 3.1.1 requires Java 17 (Java) ([#34747](https://github.com/apache/beam/issues/34747)).

Review Comment:
   Please move it under 2.66.0 Section. Here it is template.



##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumReadSchemaTransformTest.java:
##########
@@ -104,6 +105,11 @@ private PTransform<PCollectionRowTuple, PCollectionRowTuple> makePtransform(
                 // is "database.table".
                 .setTable("inventory.customers")
                 .setPort(port)
+                .setDebeziumConnectionProperties(
+                    Lists.newArrayList(
+                        "database.server.id=579676",
+                        "schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory",
+                        "schema.history.internal.file.filename=data/schema_history.dat"))

Review Comment:
   please check https://github.com/apache/beam/pull/34763/files#r2068902221.
   
   Consider use 'data/schema_history.dat' ?



##########
sdks/java/io/debezium/build.gradle:
##########
@@ -90,13 +139,25 @@ task integrationTest(type: Test, dependsOn: processTestResources) {
   }
 }
 
-configurations.all (Configuration it) -> {
-  resolutionStrategy {
-    // Force protobuf 3 because debezium is currently incompatible with protobuf 4.
-    // TODO - remove this and upgrade the version of debezium once a proto-4 compatible version is available
-    // https://github.com/apache/beam/pull/33526 does some of this, but was abandoned because it still doesn't
-    // work with protobuf 4.
-    force "com.google.protobuf:protobuf-java:3.25.5"
-    force "com.google.protobuf:protobuf-java-util:3.25.5"
-  }
+// The order is intended here - Debezium 3 requires Java 17, or later

Review Comment:
   We don't need these after `requireJavaVersion = JavaVersion.VERSION_17` added above



##########
sdks/java/io/debezium/build.gradle:
##########
@@ -38,40 +50,77 @@ dependencies {
     implementation library.java.joda_time
     provided library.java.jackson_dataformat_csv
     permitUnusedDeclared library.java.jackson_dataformat_csv
-    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
-    testImplementation project(path: ":sdks:java:io:common")
+
+    // Kafka connect dependencies
+    implementation "org.apache.kafka:connect-api:3.9.0"
+    implementation "org.apache.kafka:connect-json:3.9.0"
+    permitUnusedDeclared "org.apache.kafka:connect-json:3.9.0"
+
+    // Debezium dependencies
+    implementation group: 'io.debezium', name: 'debezium-core', version: '3.1.1.Final'
 
     // Test dependencies
+    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
+    testImplementation project(path: ":sdks:java:io:common")
     testImplementation library.java.junit
     testImplementation project(path: ":sdks:java:io:jdbc")
     testRuntimeOnly library.java.slf4j_jdk14
     testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
     testImplementation project(":runners:google-cloud-dataflow-java")
     testImplementation library.java.hamcrest
     testImplementation library.java.testcontainers_base
-    testImplementation library.java.testcontainers_mysql
-    testImplementation library.java.testcontainers_postgresql
-    // TODO(https://github.com/apache/beam/issues/31678) HikariCP 5.x requires Java11+
-    testImplementation 'com.zaxxer:HikariCP:4.0.3'
+    testImplementation "org.testcontainers:kafka"
+    testImplementation "org.testcontainers:mysql"
+    testImplementation "org.testcontainers:postgresql"
+    testImplementation "io.debezium:debezium-testing-testcontainers:3.1.1.Final"
+    testImplementation 'com.zaxxer:HikariCP:5.1.0'
 
-    // Kafka connect dependencies
-    implementation "org.apache.kafka:connect-api:2.5.0"
-    implementation "org.apache.kafka:connect-json:2.5.0"
-    permitUnusedDeclared "org.apache.kafka:connect-json:2.5.0" // BEAM-11761
+    // Debezium connector implementations for testing
+    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '3.1.1.Final'
+    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '3.1.1.Final'
+}
 
-    // Debezium dependencies
-    implementation group: 'io.debezium', name: 'debezium-core', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '1.3.1.Final'
+
+// Force Jackson versions for the test runtime classpath

Review Comment:
   add a TODO comment: remove pin after upgrading Beam's Jackson version



##########
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy:
##########
@@ -677,8 +677,8 @@ class BeamModulePlugin implements Plugin<Project> {
         activemq_junit                              : "org.apache.activemq.tooling:activemq-junit:$activemq_version",
         activemq_kahadb_store                       : "org.apache.activemq:activemq-kahadb-store:$activemq_version",
         activemq_mqtt                               : "org.apache.activemq:activemq-mqtt:$activemq_version",
-        antlr                                       : "org.antlr:antlr4:4.7",
-        antlr_runtime                               : "org.antlr:antlr4-runtime:4.7",
+        antlr                                       : "org.antlr:antlr4:4.10",
+        antlr_runtime                               : "org.antlr:antlr4-runtime:4.10",

Review Comment:
   antlr is only used by Beam SDK core and is shaded. We should be able to unpin antlr. Let me open a PR



##########
sdks/java/io/debezium/build.gradle:
##########
@@ -38,40 +50,77 @@ dependencies {
     implementation library.java.joda_time
     provided library.java.jackson_dataformat_csv
     permitUnusedDeclared library.java.jackson_dataformat_csv
-    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
-    testImplementation project(path: ":sdks:java:io:common")
+
+    // Kafka connect dependencies
+    implementation "org.apache.kafka:connect-api:3.9.0"
+    implementation "org.apache.kafka:connect-json:3.9.0"
+    permitUnusedDeclared "org.apache.kafka:connect-json:3.9.0"

Review Comment:
   why need to declare this dependency then permitUnusedDeclared?



##########
sdks/java/io/debezium/build.gradle:
##########
@@ -18,6 +18,18 @@
 import groovy.json.JsonOutput
 
 plugins { id 'org.apache.beam.module' }
+

Review Comment:
   please rebase (and possibly squash commits) onto the latest master and replace these chunk to
   
   ```
   applyJavaNature(
   ...requireJavaVersion = JavaVersion.VERSION_17
   ```



##########
sdks/java/io/debezium/expansion-service/build.gradle:
##########
@@ -20,6 +20,17 @@ apply plugin: 'org.apache.beam.module'
 apply plugin: 'application'
 mainClassName = "org.apache.beam.sdk.expansion.service.ExpansionService"
 
+// The order is intended here - Debezium 3 requires Java 17, or later

Review Comment:
   same, use `requireJavaVersion = JavaVersion.VERSION_17` and remove the version logics here and at the bottom of the file



##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOMySqlConnectorIT.java:
##########
@@ -98,19 +122,22 @@ private void monitorEssentialMetrics() {
           rs.close();
           Thread.sleep(4000);
         } else {
-          throw new IllegalArgumentException("OIOI");
+          throw new IllegalArgumentException(
+              "Illegal Argument Exception in monitorEssentialMetrics.");
         }
       }
-    } catch (InterruptedException | SQLException ex) {
-      throw new IllegalArgumentException("Oi", ex);
+    } catch (SQLException ex) {
+      LOG.error("SQL error in monitoring thread. Shutting down.", ex);

Review Comment:
   The exception is suppressed here. Is it intended? The test will pass even with exception here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2959697856

   Run Java PostCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2138786388


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -38,40 +40,83 @@ dependencies {
     implementation library.java.joda_time
     provided library.java.jackson_dataformat_csv
     permitUnusedDeclared library.java.jackson_dataformat_csv
-    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
-    testImplementation project(path: ":sdks:java:io:common")
+
+    // Kafka connect dependencies
+    implementation "org.apache.kafka:connect-api:3.9.0"
+
+    // Debezium dependencies
+    implementation group: 'io.debezium', name: 'debezium-core', version: '3.1.1.Final'
 
     // Test dependencies
+    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
+    testImplementation project(path: ":sdks:java:io:common")
     testImplementation library.java.junit
     testImplementation project(path: ":sdks:java:io:jdbc")
     testRuntimeOnly library.java.slf4j_jdk14
     testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
     testImplementation project(":runners:google-cloud-dataflow-java")
     testImplementation library.java.hamcrest
     testImplementation library.java.testcontainers_base
-    testImplementation library.java.testcontainers_mysql
-    testImplementation library.java.testcontainers_postgresql
-    // TODO(https://github.com/apache/beam/issues/31678) HikariCP 5.x requires Java11+
-    testImplementation 'com.zaxxer:HikariCP:4.0.3'
+    testImplementation "org.testcontainers:kafka"
+    testImplementation "org.testcontainers:mysql"
+    testImplementation "org.testcontainers:postgresql"
+    testImplementation "io.debezium:debezium-testing-testcontainers:3.1.1.Final"
+    testImplementation 'com.zaxxer:HikariCP:5.1.0'
 
-    // Kafka connect dependencies
-    implementation "org.apache.kafka:connect-api:2.5.0"
-    implementation "org.apache.kafka:connect-json:2.5.0"
-    permitUnusedDeclared "org.apache.kafka:connect-json:2.5.0" // BEAM-11761
+    // Debezium connector implementations for testing
+    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '3.1.1.Final'
+    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '3.1.1.Final'
+}
 
-    // Debezium dependencies
-    implementation group: 'io.debezium', name: 'debezium-core', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '1.3.1.Final'
+// TODO: Remove pin after Beam has unpinned it

Review Comment:
   Thanks, opened  #35231



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2872971976

   Hi @tkaymak, is this ready for review again? Thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2891185409

   @derrickaw
   In theory yes, but I still have failing tests in the CI that I am working on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2100445984


##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/DebeziumTransformRegistrar.java:
##########
@@ -34,7 +34,7 @@
   "nullness" // TODO(https://github.com/apache/beam/issues/20497)
 })
 public class DebeziumTransformRegistrar implements ExternalTransformRegistrar {
-  public static final String READ_JSON_URN = "beam:transform:org.apache.beam:debezium_read:v1";
+  public static final String READ_JSON_URN = "beam:transform:org.apache.beam:debezium_read:v3";

Review Comment:
   This "v1" is a URN for Beam transform. Change the name only in Java will break Python transform. No need to change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2963740570

   > Thanks!
   > 
   > We'd need to make some adjustments to CI to get DebeziumIO build and tested for cross-language tests. Let me do a separate PR, then will merge when infra is ready
   
   Need some adjustment to make PostCommit Python working. Simply bump java version would fail Spark runner tests that do not support Java 21 (yet) https://github.com/apache/beam/actions/runs/15589069843/job/43903046792
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137678930


##########
CHANGES.md:
##########
@@ -31,7 +31,7 @@
 
 ## I/Os
 
-* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Debezium IO upgraded to 3.1.1 requires Java 17 (Java) ([#34747](https://github.com/apache/beam/issues/34747)).

Review Comment:
   Thank you, done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2100611322


##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/DebeziumTransformRegistrar.java:
##########
@@ -34,7 +34,7 @@
   "nullness" // TODO(https://github.com/apache/beam/issues/20497)
 })
 public class DebeziumTransformRegistrar implements ExternalTransformRegistrar {
-  public static final String READ_JSON_URN = "beam:transform:org.apache.beam:debezium_read:v1";
+  public static final String READ_JSON_URN = "beam:transform:org.apache.beam:debezium_read:v3";

Review Comment:
   Thanks for that catch!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #34763:
URL: https://github.com/apache/beam/pull/34763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2945163822

   @Abacn it would be ready for review now, should I change the title? The failing test (Java Wordcount Direct Runner) is just flaky I think. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137678655


##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumReadSchemaTransformTest.java:
##########
@@ -104,6 +105,11 @@ private PTransform<PCollectionRowTuple, PCollectionRowTuple> makePtransform(
                 // is "database.table".
                 .setTable("inventory.customers")
                 .setPort(port)
+                .setDebeziumConnectionProperties(
+                    Lists.newArrayList(
+                        "database.server.id=579676",
+                        "schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory",
+                        "schema.history.internal.file.filename=data/schema_history.dat"))

Review Comment:
   Yes, it's now backed by a temp file, so should clean itself nicely after the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2835765171

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2068902221


##########
.gitignore:
##########
@@ -34,6 +34,9 @@ sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources
 sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/src/
 sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/sample.txt
 
+# Ignore generated debezium data
+sdks/java/io/debezium/data/

Review Comment:
   Are these files gets generated by test? We should direct temp files to `sdks/java/io/debezium/build`Otherwise these become leftovers when invoking `gradle clean`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2140238784


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -90,13 +135,8 @@ task integrationTest(type: Test, dependsOn: processTestResources) {
   }
 }
 
-configurations.all (Configuration it) -> {
-  resolutionStrategy {
-    // Force protobuf 3 because debezium is currently incompatible with protobuf 4.
-    // TODO - remove this and upgrade the version of debezium once a proto-4 compatible version is available
-    // https://github.com/apache/beam/pull/33526 does some of this, but was abandoned because it still doesn't
-    // work with protobuf 4.
-    force "com.google.protobuf:protobuf-java:3.25.5"
-    force "com.google.protobuf:protobuf-java-util:3.25.5"
-  }
+project.tasks.each {

Review Comment:
   Thank you, done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137680994


##########
sdks/java/io/debezium/build.gradle:
##########
@@ -38,40 +50,77 @@ dependencies {
     implementation library.java.joda_time
     provided library.java.jackson_dataformat_csv
     permitUnusedDeclared library.java.jackson_dataformat_csv
-    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
-    testImplementation project(path: ":sdks:java:io:common")
+
+    // Kafka connect dependencies
+    implementation "org.apache.kafka:connect-api:3.9.0"
+    implementation "org.apache.kafka:connect-json:3.9.0"
+    permitUnusedDeclared "org.apache.kafka:connect-json:3.9.0"
+
+    // Debezium dependencies
+    implementation group: 'io.debezium', name: 'debezium-core', version: '3.1.1.Final'
 
     // Test dependencies
+    testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
+    testImplementation project(path: ":sdks:java:io:common")
     testImplementation library.java.junit
     testImplementation project(path: ":sdks:java:io:jdbc")
     testRuntimeOnly library.java.slf4j_jdk14
     testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadow")
     testImplementation project(":runners:google-cloud-dataflow-java")
     testImplementation library.java.hamcrest
     testImplementation library.java.testcontainers_base
-    testImplementation library.java.testcontainers_mysql
-    testImplementation library.java.testcontainers_postgresql
-    // TODO(https://github.com/apache/beam/issues/31678) HikariCP 5.x requires Java11+
-    testImplementation 'com.zaxxer:HikariCP:4.0.3'
+    testImplementation "org.testcontainers:kafka"
+    testImplementation "org.testcontainers:mysql"
+    testImplementation "org.testcontainers:postgresql"
+    testImplementation "io.debezium:debezium-testing-testcontainers:3.1.1.Final"
+    testImplementation 'com.zaxxer:HikariCP:5.1.0'
 
-    // Kafka connect dependencies
-    implementation "org.apache.kafka:connect-api:2.5.0"
-    implementation "org.apache.kafka:connect-json:2.5.0"
-    permitUnusedDeclared "org.apache.kafka:connect-json:2.5.0" // BEAM-11761
+    // Debezium connector implementations for testing
+    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '3.1.1.Final'
+    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '3.1.1.Final'
+}
 
-    // Debezium dependencies
-    implementation group: 'io.debezium', name: 'debezium-core', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-mysql', version: '1.3.1.Final'
-    testImplementation group: 'io.debezium', name: 'debezium-connector-postgres', version: '1.3.1.Final'
+
+// Force Jackson versions for the test runtime classpath

Review Comment:
   The line before, now carries the TODO



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137677625


##########
.gitignore:
##########
@@ -34,6 +34,9 @@ sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources
 sdks/java/maven-archetypes/gcp-bom-examples/src/main/resources/archetype-resources/src/
 sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/sample.txt
 
+# Ignore generated debezium data
+sdks/java/io/debezium/data/

Review Comment:
   -> Removed, no longer needed as I switched to Temp Files



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on code in PR #34763:
URL: https://github.com/apache/beam/pull/34763#discussion_r2137678655


##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumReadSchemaTransformTest.java:
##########
@@ -104,6 +105,11 @@ private PTransform<PCollectionRowTuple, PCollectionRowTuple> makePtransform(
                 // is "database.table".
                 .setTable("inventory.customers")
                 .setPort(port)
+                .setDebeziumConnectionProperties(
+                    Lists.newArrayList(
+                        "database.server.id=579676",
+                        "schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory",
+                        "schema.history.internal.file.filename=data/schema_history.dat"))

Review Comment:
   Yes, it's now backed by a temp file, so should clean itself nicely after the test, let me know your thoughts on this please!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [IO] Update Debezium in DebeziumIO to 3.1.1 [beam]

Posted by "tkaymak (via GitHub)" <gi...@apache.org>.
tkaymak commented on PR #34763:
URL: https://github.com/apache/beam/pull/34763#issuecomment-2959707204

   I've tried to trigger a retest with 'retest this please', but it doesn't seem to be working. Could you please help trigger the tests for this PR @Abacn? The macos test seem to be flaky right now 👀 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org