You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/07/30 20:23:15 UTC

[GitHub] [pulsar] dlg99 opened a new pull request #11520: Debezium Oracle Source

dlg99 opened a new pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520


   ### Motivation
   
   Add Debezium Source for Oracle.
   
   ### Modifications
   
   Added Debezium Source for Oracle DB.
   Added integration test.
   
   The change is made on top of PRs https://github.com/apache/pulsar/pull/11154 and https://github.com/apache/pulsar/pull/11204
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
     - Added integration test
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): *yes (ojdbc)*
     - The public API: no
     - The schema: no
     - The default values of configurations: no
     - The wire protocol: no
     - The rest endpoints: no
     - The admin cli options: no
     - Anything that affects deployment: no
   
   ### Documentation
   
   #### For contributor
   
   For this PR, do we need to update docs?
   
   Need to add description of the new connector; TBD (or will create an issue later).
     
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-918794404


   @eolivelli @dlg99 The new integration test seems to be flaky. Here's one sample failure: 
   https://github.com/apache/pulsar/pull/12027/checks?check_run_id=3591596786#step:13:27471
   ```
   Error:  testDebeziumOracleDbSource(org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest)  Time elapsed: 1,800.028 s  <<< FAILURE!
   org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource() didn't finish within the time-out 1800000
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-912626746


   @sijie are you okay with merging this PR now ? it looks like licensing issues have been addressed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-905211675


   What about creating one dedicated CI job only for Oracle CDC ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] dlg99 commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-966772463


   @Anonymitaet correct, see https://github.com/apache/pulsar/pull/12213 for the docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on a change in pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#discussion_r695428164



##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"

Review comment:
       we are hard coding the version here, I am not sure that this version will be changed while doing the release
   
   In the PostGres connector we do not have this line, probably it is to be removed

##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"
+
+parallelism: 1
+
+configs:
+  ## config for Oracle XE, docker image: https://github.com/MaksymBilenko/docker-oracle-12c
+  ## docker run -d -p 1521:1521 quay.io/maksymbilenko/oracle-12c
+  database.hostname: "localhost"
+  database.port: "1521"
+  database.user: "sysdba"
+  database.password: "oracle"
+  database.dbname: "XE"
+  database.server.name: "XE"
+
+  database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"

Review comment:
       this is also to be removed or commented out

##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"
+
+parallelism: 1
+
+configs:
+  ## config for Oracle XE, docker image: https://github.com/MaksymBilenko/docker-oracle-12c
+  ## docker run -d -p 1521:1521 quay.io/maksymbilenko/oracle-12c
+  database.hostname: "localhost"

Review comment:
       should we comment out these lines? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-966771501


   Hi @dlg99 for the doc side, I think we already documented "the requirement to download and package Oracle JDBC driver" here(https://pulsar.apache.org/docs/en/next/io-debezium-source/#example-of-oracle)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] dlg99 commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-914413612


   ping @sijie 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] dlg99 commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-905048304


   @sijie @merlimat I removed dependency on Oracle JDBC and updated `tests/docker-images/latest-version-image/Dockerfile` to download ojdbc jars for the integration test. Users will have to deal with this. 
   According to https://issues.apache.org/jira/browse/LEGAL-526 FUTC is a no-go for Apache projects.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] dlg99 commented on a change in pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#discussion_r695920771



##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"
+
+parallelism: 1
+
+configs:
+  ## config for Oracle XE, docker image: https://github.com/MaksymBilenko/docker-oracle-12c
+  ## docker run -d -p 1521:1521 quay.io/maksymbilenko/oracle-12c
+  database.hostname: "localhost"
+  database.port: "1521"
+  database.user: "sysdba"
+  database.password: "oracle"
+  database.dbname: "XE"
+  database.server.name: "XE"
+
+  database.history.pulsar.service.url: "pulsar://127.0.0.1:6650"

Review comment:
       see https://github.com/apache/pulsar/pull/11520#discussion_r695920596

##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"
+
+parallelism: 1
+
+configs:
+  ## config for Oracle XE, docker image: https://github.com/MaksymBilenko/docker-oracle-12c
+  ## docker run -d -p 1521:1521 quay.io/maksymbilenko/oracle-12c
+  database.hostname: "localhost"

Review comment:
       see https://github.com/apache/pulsar/pull/11520#discussion_r695920596




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-906830691


   Thanks for your great contribution. 
   Want users to know your awesome code changes? Do not forget to add docs accordingly! And you can ping me to review the docs. Let's grow Pulsar together!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] merlimat commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
merlimat commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-893560552


   This is depending on Oracle JDBC driver https://mvnrepository.com/artifact/com.oracle.ojdbc/ojdbc8/19.3.0.0 that is licensed with "Oracle Free Use Terms and Conditions (FUTC)". It's not clear if this license is compatible with ASLv2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#issuecomment-891512678


   @dlg99 many thanks for providing doc info!
   When adding docs, you can take the following guides as references:
   - https://pulsar.apache.org/docs/en/next/io-connectors/#debezium-mysql
   - https://hub.streamnative.io/connectors/sqs-source/v2.8.0.5/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli merged pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
eolivelli merged pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] dlg99 commented on a change in pull request #11520: Debezium Oracle Source

Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #11520:
URL: https://github.com/apache/pulsar/pull/11520#discussion_r695920596



##########
File path: pulsar-io/debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+tenant: "public"
+namespace: "default"
+name: "debezium-oracle-source"
+topicName: "debezium-oracle-topic"
+archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"

Review comment:
       We do have that in multiple connectors, versions are obsolete, not sure why we have that
   ```
   $ grep -rI "archive:" . | grep -v /target/ 
   ./debezium/oracle/src/main/resources/debezium-oracle-source-config.yaml:archive: "connectors/pulsar-io-debezium-oracle-2.9.0-SNAPSHOT.nar"
   ./debezium/mongodb/src/main/resources/debezium-mongodb-source-config.yaml:archive: "connectors/pulsar-io-debezium-mongodb-2.4.0-SNAPSHOT.nar"
   ./debezium/postgres/src/main/resources/debezium-postgres-source-config.yaml:archive: "connectors/pulsar-io-debezium-postgres-2.4.0-SNAPSHOT.nar"
   ./debezium/mysql/src/main/resources/debezium-mysql-source-config.yaml:archive: "connectors/pulsar-io-debezium-mysql-2.4.0-SNAPSHOT.nar"
   ```
   
   Similarly, the config.yaml files have example configuration. 
   I don't mind removing that (as a separate change) if we agree that's the right thing to do.
   Or the release procedure/script needs update to patch the version in these configs.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org