You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/05/06 12:03:00 UTC
[GitHub] [camel-quarkus] JiriOndrusek opened a new issue #1190: Debezium MongoDB Connector native support
JiriOndrusek opened a new issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-637412595
Issue is blocked by the fact that debezium is using driver 3.x, but quarkus-mongodbd-driver is 4.x.
We have to wait for the debezium to upgrade their driver - https://issues.redhat.com/browse/DBZ-2138
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek edited a comment on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek edited a comment on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731124538
@ppalaga I have branch with tests.
Tests can be seen here: https://github.com/JiriOndrusek/camel-quarkus/tree/1190_debezium_mongodb/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/it/mongodb
All tests for debezium-* extensions are gathered together to one integration-test module.
I can review it, make sure it passes jvm and then create PR with mongodb-tests with @Disabled. WDYT?
**edited**
Sorry for not answering directly.: yes, tests are prepared and in the time of its creation, tests in jvm were successful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek edited a comment on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek edited a comment on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731124538
@ppalaga I have branch with tests.
Tests can be seen here: https://github.com/JiriOndrusek/camel-quarkus/tree/1190_debezium_mongodb/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/it/mongodb
All tests for debezium-* extensions are gathered together to one integration-test module.
I can review it, make sure it passes jvm and then create PR with mongodb-tests with @Disabled. WDYT?
**edited**
Sorry for not answering directly.: yes, test are prepared and in the time of its creation, tests in jvm were successful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731144119
I see added value with these approach. My only concern is about the fact, that all debezium integration tests are in one module and they are using common code (abstract parents).
Refactoring mongodb test into JVM module would bring redundant code (I would need to copy several classes from https://github.com/apache/camel-quarkus/tree/master/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/common/it and possibly code from https://github.com/apache/camel-quarkus/blob/master/integration-tests/debezium/src/main/java/org/apache/camel/quarkus/component/debezium/common/it/AbstractDebeziumResource.java). But because we plan to remove this jvm test and use this 'common' module, it is not probably big problem. Do you agree?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-735746210
I've started to refactor of mongodb test for jvm only. I discovered that mongodb tests are not complete in my branch (I've had probably discovered that extension was blocked before finishing them) - I'll finish them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731124538
@ppalaga I have branch with tests.
Tests can be seen here: https://github.com/JiriOndrusek/camel-quarkus/tree/1190_debezium_mongodb/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/it/mongodb
All tests for debezium-* extensions are gathered together to one integration-test module.
I can review it, make sure it passes jvm and then create PR with mongodb-tests with @Disabled. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-736529167
I'll try to build some debezium-1.4.snapshot and prepare extension to the branch.
In time debezium 1.4 is out, it should be enough to change version in pom and whole extension should be ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-739859365
Ok, issue is created - https://github.com/apache/camel-quarkus/issues/2060
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek edited a comment on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek edited a comment on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731144119
I see added value with these approach. My only concern is about the fact, that all debezium integration tests are in one module and they are using common code (abstract parents).
Refactoring mongodb test into JVM module would bring redundant code (I would need to copy several classes from https://github.com/apache/camel-quarkus/tree/master/integration-tests/debezium/src/test/java/org/apache/camel/quarkus/component/debezium/common/it and possibly code from https://github.com/apache/camel-quarkus/blob/master/integration-tests/debezium/src/main/java/org/apache/camel/quarkus/component/debezium/common/it/AbstractDebeziumResource.java). But because we plan to remove this jvm test and use this 'common' module (once native is not blocked), it is not probably big problem. Do you agree?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-736416134
It would be nice to check that Debezium master (or whichever branch they are going to release their next 1.4.x from) works for us. If so, let's wait for DBZ 1.4. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga edited a comment on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga edited a comment on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731180029
Oh, I see. OK, let's add them to the common Debezium test module and let's not create the native test class so that they run only in the JVM mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek edited a comment on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek edited a comment on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-637412595
Issue is blocked by the fact that debezium is using driver 3.x, but quarkus-mongodb-client uses 4.x.
We have to wait for the debezium to upgrade their driver - https://issues.redhat.com/browse/DBZ-2138
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-624608876
Please, assign to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731180029
Oh, I see. OK, let's add them to the common Debezium test module and let's not create the native test class so that they run only in the native mode?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga closed issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga closed issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-736320968
After investigation it seems that it is not possible to have working jvm tests for debezium mongodb. Here are reasons:
- Camel-quarkus uses mongodb version from quarkus. Which brings driver of version **4.x** - https://github.com/apache/camel-quarkus/commit/d07e2a5ec7dc964c660dde3d7edc2cc1a3be9fdc#diff-2cdf2d35793fae7cd8f70171d04de916f2a6c295b19f099a0aea50589be615b4R442
- Connection to mongodb from component is provided by `io.debezium:debezium-connector-mongodb:jar:1.3.0.Final`, which will support driver 4.x from version 1.4 (https://issues.redhat.com/browse/DBZ-2138)
Moreover current debezium mongodb extension does not work in JVM, because it uses newer mongodb driver.
Solution could be to exclude all new mongodb dependencies and use old ones. But because versions of such dependencies are restricted by bom. Solution, where there are more versions of mongodb driver in bom, should not be possible. (In other words, making jvm debeziom-mongodbwork, would fail all extension using mongdb)
@ppalaga What do you think? I'd suggest to wait with completing extension to release of `io.debezium:debezium-connector-mongodb:jar:1.4..x`. It would be also nice to remove debezium-mongodb from extension-jvm, because it can not work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-740708733
> 'll look into an option to make it work with 1.3.0-final version of debezium-connector
Do not hesitate to give up if it turns out to be too much work. DBZ 1.4.x might come soon enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] jamesnetherton commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
jamesnetherton commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-736531668
> In time debezium 1.4 is out, it should be enough to change version in pom and whole extension should be ready.
We should bump the Debezium version on the Camel side too. Which may actually be a prerequisite to getting this working. I remember in the past we had issues where code was compiled against the Mongo 3.x drivers but 4.x was used at runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-739842959
I'm trying to prepare JVM tests for debezium-mongodb. But I've found a problem in debezium-connector, which prevents me to finish test using test container (and therefore random port). I've reported issue as https://issues.redhat.com/browse/DBZ-2831.
After the fix, I'll continue with preparation of tests using 1.4.0-SNAPSOT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-739849885
Thanks @JiriOndrusek, DBZ-2831 being an issue with quite a serious impact, could you please file a proxy issue on our side too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-740689166
@ppalaga I've created jvm only test in extension-jvm/integration-test in my repo - https://github.com/JiriOndrusek/camel-quarkus/tree/debezium-mongodb-test-jvm-attempt-2
Test is using 1.4.0-SNAPSHOT of debezium-connector - it can not be merged as a part of jvm component so far.
If mongodb container's exposed port is set to 27017 (default one) - [code](https://github.com/apache/camel-quarkus/compare/master...JiriOndrusek:debezium-mongodb-test-jvm-attempt-2#diff-7db91bac166524ae6549f31ada1cf2206fdde99e4f90659377014b97866555a5R51) , test is successful.
I'll look into an option to make it work with 1.3.0-final version of debezium-connector to be able to marge it into git...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-736637975
Yes, upgrade of debezium in camel sounds like a correct step. I'll create jira ticket for it and link here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731121341
@JiriOndrusek do you have a proper test and is it passing the JVM mode at least?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] JiriOndrusek commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
JiriOndrusek commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731185521
ok, sound good. I'll prepare it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-quarkus] ppalaga commented on issue #1190: Debezium MongoDB Connector native support
Posted by GitBox <gi...@apache.org>.
ppalaga commented on issue #1190:
URL: https://github.com/apache/camel-quarkus/issues/1190#issuecomment-731139526
I vote for adding the tests to the existing JVM module https://github.com/apache/camel-quarkus/tree/master/extensions-jvm/debezium-mongodb/integration-test
They can well be enabled there and increase our confidence that the extension works well in the JVM mode. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org