You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "devinbost (via GitHub)" <gi...@apache.org> on 2023/03/15 11:30:30 UTC

[GitHub] [pulsar] devinbost opened a new pull request, #19821: Added Debezium DB2 connector for Pulsar

devinbost opened a new pull request, #19821:
URL: https://github.com/apache/pulsar/pull/19821

   Fixes #7837
   Adds DB2 connector to pulsar-io
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1152138419


##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumDB2DbContainer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.containers;
+
+
+import java.time.Duration;
+import java.time.temporal.ChronoUnit;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+
+public class DebeziumDB2DbContainer extends ChaosContainer<DebeziumDB2DbContainer> {
+
+    public static final String NAME = "debezium-db2";
+    static final Integer[] PORTS = { 50000 };
+
+    //
+    private static final String IMAGE_NAME = "devingbost/debezium-db2:0.0.1";

Review Comment:
   I built the image from this public Dockerfile: https://github.com/debezium/debezium-connector-db2/blob/bee92787b1e02bb3e0813a6d8b7f62d34ad6d9e2/src/test/docker/db2-cdc-docker/Dockerfile#L1
   However, I was unable to find any Debezium image available publically using this Dockerfile, so I built my own. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [wip][feat][io] Debezium DB2 source connector for Pulsar [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar
URL: https://github.com/apache/pulsar/pull/19821


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1152138419


##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumDB2DbContainer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.containers;
+
+
+import java.time.Duration;
+import java.time.temporal.ChronoUnit;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+
+public class DebeziumDB2DbContainer extends ChaosContainer<DebeziumDB2DbContainer> {
+
+    public static final String NAME = "debezium-db2";
+    static final Integer[] PORTS = { 50000 };
+
+    //
+    private static final String IMAGE_NAME = "devingbost/debezium-db2:0.0.1";

Review Comment:
   I built the image from this public Dockerfile: https://github.com/debezium/debezium-connector-db2/blob/bee92787b1e02bb3e0813a6d8b7f62d34ad6d9e2/src/test/docker/db2-cdc-docker/Dockerfile#L1
   However, I was unable to find any Debezium image available publically using this Dockerfile, so I built my own. I'm open to suggestions here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dave2wave commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "dave2wave (via GitHub)" <gi...@apache.org>.
dave2wave commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1637175947

   @devinbost @eolivelli It looks like this is close, but now there are CI and TestContainer updates needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1635163767

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1589299655

   It's very close to being finished but I just haven't had time to finish it.
   There's just a test that needs to pass. IIRC there was a mismatch in the
   number of records the connector was emitting and the number expected due to
   something being emitted twice.
   
   --
   Devin Bost
   Sent from mobile
   
   On Mon, Jun 12, 2023, 7:57 PM tison ***@***.***> wrote:
   
   > cc @devinbost <https://github.com/devinbost> @eolivelli
   > <https://github.com/eolivelli> do we still make progress on this PR?
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/pulsar/pull/19821#issuecomment-1588342461>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ABYTBL3MUEGT24YKV3RYSODXK63H5ANCNFSM6AAAAAAV3VRUT4>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1151118274


##########
pulsar-io/debezium/db2/src/main/resources/debezium-db2-source-config.yaml:
##########
@@ -0,0 +1,37 @@
+#

Review Comment:
   I ensured it's up-to-date with the doc. Many of the other connectors have this file. 
   <img width="669" alt="image" src="https://user-images.githubusercontent.com/7418031/228358769-f8c6f9ac-f366-4200-881d-96c082c7139c.png">
   
   If we want to remove it, we may want to discuss removing the others on the mailing list. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dlg99 commented on a diff in pull request #19821: [WIP] Added Debezium DB2 connector for Pulsar

Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1148169197


##########
pulsar-io/debezium/db2/src/main/resources/debezium-db2-source-config.yaml:
##########
@@ -0,0 +1,37 @@
+#

Review Comment:
   TBH I do not remember. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1153721359


##########
tests/docker-images/latest-version-image/Dockerfile:
##########
@@ -129,5 +129,16 @@ RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remo
 
 RUN jar uf connectors/pulsar-io-debezium-oracle-*.nar META-INF/bundled-dependencies/ojdbc8-19.3.0.0.jar META-INF/bundled-dependencies/ucp-19.3.0.0.jar META-INF/bundled-dependencies/oraclepki-19.3.0.0.jar META-INF/bundled-dependencies/osdt_cert-19.3.0.0.jar META-INF/bundled-dependencies/osdt_core-19.3.0.0.jar META-INF/bundled-dependencies/simplefan-19.3.0.0.jar META-INF/bundled-dependencies/orai18n-19.3.0.0.jar META-INF/bundled-dependencies/xdb-19.3.0.0.jar META-INF/bundled-dependencies/xmlparserv2-19.3.0.0.jar
 
-
+# Download IBM DB2 JDBC driver for DB2 Debezium Connector tests:
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/11.5.8.0/jcc-11.5.8.0.jar
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/db2jcc/db2jcc4/db2jcc-db2jcc4.jar

Review Comment:
   Additionally, "Client license files are not required for connections to Db2, Cloudscape, or IBM Informix® databases from the IBM Data Server Driver for JDBC and SQLJ version 3.50 or later. Client license files are not required for direct connections to Db2 for z/OS if Db2 Connect Unlimited Edition for System z® server license activation is performed."
   (https://www.ibm.com/docs/en/db2/11.5?topic=apis-installing-data-server-driver-jdbc-sqlj)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] tisonkun commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1588342461

   cc @devinbost @eolivelli do we still make progress on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [wip][feat][io] Debezium DB2 source connector for Pulsar [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-6179024775

   Thanks for this contribution. The IO connectors have moved out of this repository to https://github.com/apache/pulsar-connectors (PIP-465), so this PR can no longer be merged here. If the change is still needed, please open it against apache/pulsar-connectors. This is part of the open-PR cleanup discussed on dev@pulsar.apache.org.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1700228099

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #19821: Added Debezium DB2 connector for Pulsar

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1469837310

   @devinbost Please add the following content to your PR description and select a checkbox:
   ```
   - [ ] `doc` <!-- Your PR contains doc changes -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] Anonymitaet commented on pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "Anonymitaet (via GitHub)" <gi...@apache.org>.
Anonymitaet commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1487862437

   Hi @devinbost, I see you've contributed docs along with code changes, thank you!
   
   A gentle reminder: have you submitted a PIP for this change? Can you attach the PIP vote link in the PR description?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] nicoloboschi commented on a diff in pull request #19821: [WIP] Added Debezium DB2 connector for Pulsar

Posted by "nicoloboschi (via GitHub)" <gi...@apache.org>.
nicoloboschi commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1141758258


##########
pulsar-io/debezium/db2/src/main/resources/debezium-db2-source-config.yaml:
##########
@@ -0,0 +1,37 @@
+#

Review Comment:
   what is the purpose of this file? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dlg99 commented on a diff in pull request #19821: [WIP] Added Debezium DB2 connector for Pulsar

Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1146986919


##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumDB2DbSourceTester.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.io.sources.debezium;
+
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import lombok.Getter;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.pulsar.tests.integration.containers.DebeziumDB2DbContainer;
+import org.apache.pulsar.tests.integration.containers.DebeziumOracleDbContainer;
+import org.apache.pulsar.tests.integration.containers.PulsarContainer;
+import org.apache.pulsar.tests.integration.docker.ContainerExecResult;
+import org.apache.pulsar.tests.integration.io.sources.SourceTester;
+import org.apache.pulsar.tests.integration.topologies.PulsarCluster;
+import org.testng.util.Strings;
+
+/**
+ * A tester for testing Debezium OracleDb source.
+ */
+@Slf4j
+public class DebeziumDB2DbSourceTester extends SourceTester<DebeziumDB2DbContainer> {
+
+    private static final String NAME = "debezium-db2";
+    private static final long SLEEP_AFTER_COMMAND_MS = 30_000;
+
+    private final String pulsarServiceUrl;
+
+    @Getter
+    private DebeziumDB2DbContainer debeziumDB2DbContainer;
+
+    private final PulsarCluster pulsarCluster;
+
+    public DebeziumDB2DbSourceTester(PulsarCluster cluster) {
+        super(NAME);
+        this.pulsarCluster = cluster;
+        this.numEntriesToInsert = 1;
+        this.numEntriesExpectAfterStart = 0;
+
+        pulsarServiceUrl = "pulsar://pulsar-proxy:" + PulsarContainer.BROKER_PORT;
+        sourceConfig.put("connector.class", "io.debezium.connector.db2.Db2Connector");
+        sourceConfig.put("database.hostname", DebeziumDB2DbContainer.NAME);
+        sourceConfig.put("database.port", "50000");
+        sourceConfig.put("database.user", "db2inst1");
+        sourceConfig.put("database.password", "admin");
+        sourceConfig.put("database.dbname", "mydb2");
+        sourceConfig.put("topic.prefix", "stores");
+        sourceConfig.put("table.include.list", "DB2INST1.STORES");
+
+        //sourceConfig.put("snapshot.mode", "schema_only");  //??
+
+        //sourceConfig.put("schema.include.list", "inv"); //??

Review Comment:
   please remove



##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumDB2DbContainer.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.containers;
+
+
+import java.time.Duration;
+import java.time.temporal.ChronoUnit;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+
+public class DebeziumDB2DbContainer extends ChaosContainer<DebeziumDB2DbContainer> {
+
+    public static final String NAME = "debezium-db2";
+    static final Integer[] PORTS = { 50000 };
+
+    //
+    private static final String IMAGE_NAME = "devingbost/debezium-db2:0.0.1";
+
+    public DebeziumDB2DbContainer(String clusterName) {
+        super(clusterName, IMAGE_NAME);
+    }
+
+    @Override
+    public String getContainerName() {
+        return clusterName;
+    }
+
+    @Override
+    protected void configure() {
+        super.configure();
+        this.withNetworkAliases(NAME)
+            .withExposedPorts(PORTS)
+                .withEnv("LICENSE", "accept")
+                .withEnv("DB2INSTANCE", "db2inst1")
+                .withEnv("DB2INST1_PASSWORD", "admin")
+                .withEnv("DBNAME", "mydb2")
+                .withEnv("BLU", "false")
+                .withEnv("ENABLE_ORACLE_COMPATIBILITY", "false")
+                .withEnv("UPDATEAVAIL", "NO")
+                .withEnv("TO_CREATE_SAMPLEDB", "false")
+                .withEnv("REPODB", "false")
+                .withEnv("IS_OSXFS", "false")
+                .withEnv("PERSISTENT_HOME", "true")
+                .withEnv("HADR_ENABLED", "false")
+               // .withEnv("ETCD_ENDPOINT", "")
+               // .withEnv("ETCD_USERNAME", "")
+               // .withEnv("ETCD_PASSWORD", "")

Review Comment:
   please remove



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dlg99 commented on pull request #19821: [WIP] Added Debezium DB2 connector for Pulsar

Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1483622709

   Please follow up with docs update https://github.com/apache/pulsar-site/blob/main/docs/io-debezium-source.md


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [WIP] Added Debezium DB2 connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1147374231


##########
pulsar-io/debezium/db2/src/main/resources/debezium-db2-source-config.yaml:
##########
@@ -0,0 +1,37 @@
+#

Review Comment:
   @dlg99 Do you know? The other connectors had it, so I included it as well. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] eolivelli commented on pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1489824677

   I don't think that a PIP is required for a new Connector.
   
   PIP are important for new APIs and for API changes/breaking changes.
   
   Add a new Connector has very little impact, especially for this kind of connectors


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] lhotari commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1151470087


##########
pulsar-io/debezium/db2/src/main/java/org/apache/pulsar/io/debezium/db2/package-info.java:
##########
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.debezium.oracle;

Review Comment:
   ```suggestion
   package org.apache.pulsar.io.debezium.db2;
   ```



##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumDB2DbContainer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.containers;
+
+
+import java.time.Duration;
+import java.time.temporal.ChronoUnit;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+
+public class DebeziumDB2DbContainer extends ChaosContainer<DebeziumDB2DbContainer> {
+
+    public static final String NAME = "debezium-db2";
+    static final Integer[] PORTS = { 50000 };
+
+    //
+    private static final String IMAGE_NAME = "devingbost/debezium-db2:0.0.1";

Review Comment:
   Is there a public image available that could be used instead? How is this image maintained?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dlg99 commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1152284135


##########
tests/docker-images/latest-version-image/Dockerfile:
##########
@@ -129,5 +129,16 @@ RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remo
 
 RUN jar uf connectors/pulsar-io-debezium-oracle-*.nar META-INF/bundled-dependencies/ojdbc8-19.3.0.0.jar META-INF/bundled-dependencies/ucp-19.3.0.0.jar META-INF/bundled-dependencies/oraclepki-19.3.0.0.jar META-INF/bundled-dependencies/osdt_cert-19.3.0.0.jar META-INF/bundled-dependencies/osdt_core-19.3.0.0.jar META-INF/bundled-dependencies/simplefan-19.3.0.0.jar META-INF/bundled-dependencies/orai18n-19.3.0.0.jar META-INF/bundled-dependencies/xdb-19.3.0.0.jar META-INF/bundled-dependencies/xmlparserv2-19.3.0.0.jar
 
-
+# Download IBM DB2 JDBC driver for DB2 Debezium Connector tests:
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/11.5.8.0/jcc-11.5.8.0.jar
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/db2jcc/db2jcc4/db2jcc-db2jcc4.jar

Review Comment:
   Please double-check that IBM's license is ok with db2jcc use for testing without purchase of db2 license.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #19821: [wip][feat][io] Debezium DB2 source connector for Pulsar

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1547085367

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] eolivelli commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1152844022


##########
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumDB2DbContainer.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.tests.integration.containers;
+
+
+import java.time.Duration;
+import java.time.temporal.ChronoUnit;
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy;
+
+public class DebeziumDB2DbContainer extends ChaosContainer<DebeziumDB2DbContainer> {
+
+    public static final String NAME = "debezium-db2";
+    static final Integer[] PORTS = { 50000 };
+
+    //
+    private static final String IMAGE_NAME = "devingbost/debezium-db2:0.0.1";

Review Comment:
   I don't think that we can use this docker image, as a ASF OSS project we should only reference well established public sources for the code.
   
   If this is not possible then I am leaning toward asking to not implement the integration test at all.
   We can say that it is it tested manually.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] eolivelli commented on pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1489827485

   I am generally +1 on this work, but I left a comment about the tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on a diff in pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on code in PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#discussion_r1153696673


##########
tests/docker-images/latest-version-image/Dockerfile:
##########
@@ -129,5 +129,16 @@ RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remo
 
 RUN jar uf connectors/pulsar-io-debezium-oracle-*.nar META-INF/bundled-dependencies/ojdbc8-19.3.0.0.jar META-INF/bundled-dependencies/ucp-19.3.0.0.jar META-INF/bundled-dependencies/oraclepki-19.3.0.0.jar META-INF/bundled-dependencies/osdt_cert-19.3.0.0.jar META-INF/bundled-dependencies/osdt_core-19.3.0.0.jar META-INF/bundled-dependencies/simplefan-19.3.0.0.jar META-INF/bundled-dependencies/orai18n-19.3.0.0.jar META-INF/bundled-dependencies/xdb-19.3.0.0.jar META-INF/bundled-dependencies/xmlparserv2-19.3.0.0.jar
 
-
+# Download IBM DB2 JDBC driver for DB2 Debezium Connector tests:
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/11.5.8.0/jcc-11.5.8.0.jar
+RUN cd META-INF/bundled-dependencies && curl -sSLO https://search.maven.org/remotecontent?filepath=com/ibm/db2/jcc/db2jcc/db2jcc4/db2jcc-db2jcc4.jar

Review Comment:
   The IBM InfoSphere CDC and Data Replication license authorizes non-production use, as specified below:
   "If the Program is designated as "Non-Production", the Program can only be deployed as part of the Licensee's internal development and test environment for internal non-production activities, including but not limited to testing, performance tuning, fault diagnosis, internal benchmarking, staging, quality assurance activity and/or developing internally used additions or extensions to the Program using published application programming interfaces. Licensee is not authorized to use any part of the Program for any other purposes without acquiring the appropriate production entitlements."
   (https://www.ibm.com/support/customer/csol/terms/?id=L-GMET-8KJN5X&lc=en#detail-document)
   
   Is there a place I should put this text in my PR? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] devinbost commented on pull request #19821: [feat][io] Debezium DB2 source connector for Pulsar

Posted by "devinbost (via GitHub)" <gi...@apache.org>.
devinbost commented on PR #19821:
URL: https://github.com/apache/pulsar/pull/19821#issuecomment-1488860517

   > A gentle reminder: have you submitted a PIP for this change? Can you attach the PIP vote link in the PR description?
   
   Hi @Anonymitaet , I don't think a PIP is typically required for new connectors. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org