You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/01/21 08:10:14 UTC
[GitHub] [pulsar] huangdx0726 opened a new pull request #6112: [Issue 6111]
Add Debezium Mongodb for io-connectors
huangdx0726 opened a new pull request #6112: [Issue 6111] Add Debezium Mongodb for io-connectors
URL: https://github.com/apache/pulsar/pull/6112
Fixes #6111
### Motivation
Add Debezium Mongodb for io-connectors .
### Modifications
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #6112: [Issue
6111] Add Debezium Mongodb for io-connectors
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #6112: [Issue 6111] Add Debezium Mongodb for io-connectors
URL: https://github.com/apache/pulsar/pull/6112#discussion_r369459465
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -1,14 +1,15 @@
---
-id: io-debezium-source
-title: Debezium source onnector
+id: io-cdc-debezium
Review comment:
why do you change this id? This is going to mess up all the documentation links. I would prefer not changing any doc id.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie merged pull request #6112: [Issue 6111][Doc] Add
documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #6112: [Issue
6111] Add Debezium Mongodb for io-connectors
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #6112: [Issue 6111] Add Debezium Mongodb for io-connectors
URL: https://github.com/apache/pulsar/pull/6112#discussion_r369460253
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -27,6 +28,13 @@ The configuration of the Debezium source connector has the following properties.
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `pulsar.service.url` | true | null | Pulsar cluster service URL. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
+| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). |
Review comment:
if these settings are only used for mongodb, please a subsection under `documentation` for mongodb specific settings.
```
### MongoDB Configuration
| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). |
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] huangdx0726 opened a new pull request #6112: [Issue
6111][Doc] Add documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
huangdx0726 opened a new pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112
Fixes #6111
### Motivation
Add Debezium Mongodb for io-connectors .
### Modifications
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] tuteng commented on issue #6112: [Issue 6111] Add Debezium
Mongodb for io-connectors
Posted by GitBox <gi...@apache.org>.
tuteng commented on issue #6112: [Issue 6111] Add Debezium Mongodb for io-connectors
URL: https://github.com/apache/pulsar/pull/6112#issuecomment-576976296
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] tuteng closed pull request #6112: [Issue 6111][Doc] Add
documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
tuteng closed pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] huangdx0726 commented on a change in pull request #6112:
[Issue 6111][Doc] Add documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on a change in pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112#discussion_r375087955
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -1,14 +1,15 @@
---
-id: io-debezium-source
-title: Debezium source onnector
+id: io-cdc-debezium
Review comment:
I'll fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #6112: [Issue
6111] Add Debezium Mongodb for io-connectors
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #6112: [Issue 6111] Add Debezium Mongodb for io-connectors
URL: https://github.com/apache/pulsar/pull/6112#discussion_r369460525
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -304,7 +312,124 @@ This example shows how to change the data of a PostgreSQL table using the Pulsar
----- got message -----
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":107}}�{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"db"},{"type":"int64","optional":true,"field":"ts_usec"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"string","optional":true,"field":"schema"},{"type":"string","optional":true,"field":"table"},{"type":"boolean","optional":true,"default":false,"field":"snapshot"},{"type":"boolean","optional":true,"field":"last_snapshot_record"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"before":{"id":107,"name":"rocks","description":"box of assorted rocks","weight":5.3},"after":{"id":107,"name":"1111111111","description":"box of assorted rocks","weight":5.3},"source":{"version":"0.9.2.Final","connector":"postgresql","name":"dbserver1","db":"postgres","ts_usec":1559208957661080,"txId":577,"lsn":23862872,"schema":"inventory","table":"products","snapshot":false,"last_snapshot_record":null},"op":"u","ts_ms":1559208957692}}
```
+## Example of MongoDB
+
+You need to create a configuration file before using the Pulsar Debezium connector.
+
+* JSON
+
+ ```json
+ {
+ "mongodb.hosts": "rs0/mongodb:27017",
+ "mongodb.name": "dbserver1",
+ "mongodb.user": "debezium",
+ "mongodb.password": "dbz",
+ "mongodb.task.id": "1",
+ "database.whitelist": "inventory",
+ "pulsar.service.url": "pulsar://127.0.0.1:6650"
+ }
+ ```
+
+* YAML
+
+ You can create a `debezium-mongodb-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mongodb/src/main/resources/debezium-mongodb-source-config.yaml) below to the `debezium-mongodb-source-config.yaml` file.
+
+ ```yaml
+ tenant: "public"
+ namespace: "default"
+ name: "debezium-mongodb-source"
+ topicName: "debezium-mongodb-topic"
+ archive: "connectors/pulsar-io-debezium-mongodb-{{pulsar:version}}.nar"
+ parallelism: 1
+
+ configs:
+
+ ## config for pg, docker image: debezium/example-postgress:0.10
Review comment:
This comment should be updated to mongodb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] huangdx0726 commented on a change in pull request #6112:
[Issue 6111][Doc] Add documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on a change in pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112#discussion_r376861920
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -1,14 +1,15 @@
---
-id: io-debezium-source
-title: Debezium source onnector
+id: io-cdc-debezium
Review comment:
I have fixed it,PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] huangdx0726 commented on a change in pull request #6112:
[Issue 6111][Doc] Add documentation for Debezium Mongodb connector
Posted by GitBox <gi...@apache.org>.
huangdx0726 commented on a change in pull request #6112: [Issue 6111][Doc] Add documentation for Debezium Mongodb connector
URL: https://github.com/apache/pulsar/pull/6112#discussion_r376861879
##########
File path: site2/docs/io-debezium-source.md
##########
@@ -27,6 +28,13 @@ The configuration of the Debezium source connector has the following properties.
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `pulsar.service.url` | true | null | Pulsar cluster service URL. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
+| `mongodb.hosts` | true | null | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017). |
Review comment:
I have fixed it,PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services