You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/11/22 01:09:24 UTC
[GitHub] [pulsar] tuteng commented on a change in pull request #5715:
[Issues 5692]Add document for mongodb connector
tuteng commented on a change in pull request #5715: [Issues 5692]Add document for mongodb connector
URL: https://github.com/apache/pulsar/pull/5715#discussion_r349390966
##########
File path: site2/docs/io-cdc-debezium.md
##########
@@ -305,3 +312,114 @@ This example shows how to change the data of a PostgreSQL table using the Pulsar
----- got message -----
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"optional":false,"name":"dbserver1.inventory.products.Key"},"payload":{"id":107}}�{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":true,"field":"description"},{"type":"double","optional":true,"field":"weight"}],"optional":true,"name":"dbserver1.inventory.products.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":true,"field":"version"},{"type":"string","optional":true,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"string","optional":false,"field":"db"},{"type":"int64","optional":true,"field":"ts_usec"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"string","optional":true,"field":"schema"},{"type":"string","optional":true,"field":"table"},{"type":"boolean","optional":true,"default":false,"field":"snapshot"},{"type":"boolean","optional":true,"field":"last_snapshot_record"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"dbserver1.inventory.products.Envelope"},"payload":{"before":{"id":107,"name":"rocks","description":"box of assorted rocks","weight":5.3},"after":{"id":107,"name":"1111111111","description":"box of assorted rocks","weight":5.3},"source":{"version":"0.9.2.Final","connector":"postgresql","name":"dbserver1","db":"postgres","ts_usec":1559208957661080,"txId":577,"lsn":23862872,"schema":"inventory","table":"products","snapshot":false,"last_snapshot_record":null},"op":"u","ts_ms":1559208957692}}
```
+## Example of MongoDB
+
+You need to create a configuration file before using the Pulsar Debezium connector.
+
+* JSON
+
+ ```json
+ {
+ "mongodb.hosts": "rs0/mongodb:27017",
+ "mongodb.name": "dbserver1",
+ "mongodb.user": "debezium",
+ "mongodb.password": "dbz",
+ "mongodb.task.id": "1",
+ "schema.whitelist": "inventory",
+ "pulsar.service.url": "pulsar://127.0.0.1:6650"
+ }
+ ```
+
+* YAML
+
+ You can create a `debezium-postgres-source-config.yaml` file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/resources/debezium-postgres-source-config.yaml) below to the `debezium-postgres-source-config.yaml` file.
+
+ ```yaml
+ tenant: "public"
+ namespace: "default"
+ name: "debezium-mongodb-source"
+ topicName: "debezium-mongodb-topic"
+ archive: "connectors/pulsar-io-debezium-mongodb-{{pulsar:version}}.nar"
+ parallelism: 1
+
+ configs:
+
+ ## config for pg, docker image: debezium/example-postgress:0.8
+ "mongodb.hosts": "rs0/mongodb:27017",
+ "mongodb.name": "dbserver1",
+ "mongodb.user": "debezium",
+ "mongodb.password": "dbz",
+ "mongodb.task.id": "1",
+ "schema.whitelist": "inventory",
+
+ ## PULSAR_SERVICE_URL_CONFIG
+ pulsar.service.url: "pulsar://127.0.0.1:6650"
+ ```
+
+### Usage
+
+This example shows how to change the data of a MongoDB table using the Pulsar Debezium connector.
+
+
+1. Start a PostgreSQL server with a database from which Debezium can capture changes.
+
+ ```bash
+ $ docker pull debezium/example-mongodb:0.10
+ $ docker run -d -it --rm --name pulsar-mongodb -e MONGODB_USER=mongodb -e MONGODB_PASSWORD=mongodb -p 27017:27017 debezium/example-mongodb:0.10
+ ```
+
+2. Start a Pulsar service locally in standalone mode.
+
+ ```bash
+ $ bin/pulsar standalone
+ ```
+
+3. Start the Pulsar Debezium connector in local run mode using one of the following methods.
+
+ * Use the **JSON** configuration file as shown previously.
+
+ Make sure the nar file is available at `connectors/pulsar-io-mongodb-postgres-{{pulsar:version}}.nar`.
+
+ ```bash
+ $ bin/pulsar-admin source localrun \
+ --archive connectors/pulsar-io-debezium-postgres-{{pulsar:version}}.nar \
+ --name debezium-mongodb-source \
+ --destination-topic-name debezium-mongodb-topic \
+ --tenant public \
+ --namespace default \
+ --source-config '{"mongodb.hosts": "rs0/mongodb:27017","mongodb.name": "dbserver1","mongodb.user": "debezium","mongodb.password": "dbz","mongodb.task.id": "1","schema.whitelist": "inventory","pulsar.service.url": "pulsar://127.0.0.1:6650"}'
+ ```
+
+ * Use the **YAML** configuration file as shown previously.
+
+ ```bash
+ $ bin/pulsar-admin source localrun \
+ --source-config-file debezium-mongodb-source-config.yaml
+ ```
+
+4. Subscribe the topic _sub-products_ for the _inventory.products_ table.
+
+ ```
+ $ bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0
+ ```
+
+5. Start a MongoDB client in docker.
+
+ ```bash
+ $ docker exec -it pulsar-mongodb /bin/bash
+ ```
+
+6. A MongoDB client pops out.
+
+ Use the following commands to init the data.
+
+ ``` bash
+ ./usr/local/bin/init-inventory.sh
Review comment:
I think this should be done after the database initialization is completed because the replication set should be initialized first. After initializing the replication set, you need to add a DNS resolution of the container name under your /etc/hosts file, otherwise, the connector cannot find this address. After the script is initialized, new permissions need to be added. In order to configure the replication set, root permission can be temporarily used for testing. You can view the status and configuration of replication sets through `rs.status()` and `rs.conf()`.
Update permissions:
```
docker exec -it pulsar-mongodb /bin/bash
mongo -u admin -p admin localhost:27017/admin
db.updateUser('admin', { pwd: 'admin', roles: [ {role: "root", db: "admin" } ] });
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services