You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/28 06:54:29 UTC
[GitHub] [pulsar] eav opened a new issue #8737: pulsar-admin doesn't respect pulsar.service.url set in the connector config
eav opened a new issue #8737:
URL: https://github.com/apache/pulsar/issues/8737
**Describe the bug**
Trying to run debezium-postgres connector from pod in kubernetes, pg and pulsar are also deployed as separate pods. Looks like connector doesn't take into account provided pulsar.service.url: "pulsar://pulsar:6650" but trying to connect to localhost/127.0.0.1:6650
The run command for connector:
```
command: ["./bin/pulsar-admin", "sources", "localrun", "--source-config-file", "configs/pg-pulsar-config.yaml"]
```
Logs from connector:
```
06:29:50.783 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to localhost:6650 : io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:6650
06:30:20.010 [function-timer-thread-6-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/debezium-postgres-source-java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:6650 Function Container is dead with exception.. restarting
06:30:20.011 [function-timer-thread-6-1] INFO org.apache.pulsar.functions.runtime.thread.ThreadRuntime - ThreadContainer starting function with instance config InstanceConfig(instanceId=0, functionId=21d2e125-382f-4da7-ab79-36eccfe8966d, functionVersion=84fca34b-91ad-4075-abd8-5f44182d1250, functionDetails=tenant: "public"
namespace: "default"
name: "debezium-postgres-source"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
autoAck: true
parallelism: 1
source {
className: "org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource"
configs: "{\"database.user\":\"postgres\",\"database.dbname\":\"postgres\",\"database.hostname\":\"pg\",\"database.password\":\"postgres\",\"database.server.name\":\"dbserver1\",\"database.port\":\"5432\",\"pulsar.service.url\":\"pulsar://pulsar:6650\"}"
typeClassName: "org.apache.pulsar.common.schema.KeyValue"
}
sink {
topic: "debezium-postgres-topic"
typeClassName: "org.apache.pulsar.common.schema.KeyValue"
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SOURCE
, maxBufferedTuples=1024, functionAuthenticationSpec=null, port=33649, clusterName=local, maxPendingAsyncRequests=1000)
06:30:20.786 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to localhost:6650 : io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:6650
```
Connector config:
```
tenant: "public"
namespace: "default"
name: "debezium-postgres-source"
topicName: "debezium-postgres-topic"
archive: "connectors/pulsar-io-debezium-postgres-2.6.2.nar"
parallelism: 1
configs:
database.hostname: "pg"
database.port: "5432"
database.user: "postgres"
database.password: "postgres"
database.dbname: "postgres"
database.server.name: "dbserver1"
pulsar.service.url: "pulsar://pulsar:6650"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai closed issue #8737: pulsar-admin doesn't respect pulsar.service.url set in the connector config
Posted by GitBox <gi...@apache.org>.
jiazhai closed issue #8737:
URL: https://github.com/apache/pulsar/issues/8737
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Huanli-Meng commented on issue #8737: pulsar-admin doesn't respect pulsar.service.url set in the connector config
Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on issue #8737:
URL: https://github.com/apache/pulsar/issues/8737#issuecomment-735626121
Got it. will check the doc and update it accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eav commented on issue #8737: pulsar-admin doesn't respect pulsar.service.url set in the connector config
Posted by GitBox <gi...@apache.org>.
eav commented on issue #8737:
URL: https://github.com/apache/pulsar/issues/8737#issuecomment-735716795
ok, works now, I had to provide --broker-service-url pulsar://pulsar:6650 as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #8737: pulsar-admin doesn't respect pulsar.service.url set in the connector config
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #8737:
URL: https://github.com/apache/pulsar/issues/8737#issuecomment-735624642
`pulsar.service.url` is used for the offset topic used in debezium. It is not the one used for connecting to the sink topic.
You can run `bin/pulsar-admin --admin-url http://pulsar:8080 sources localrun --source-config-file configs/pg-pulsar-config.yaml` to point to the right pulsar cluster.
---
@Jennifer88huang @Huanli-Meng Can you take a look at this thread to improve our documentation for debezium connector?
@wolfstudy I think we can improve debezium connector to use same pulsar url for both the history topic used by the debezium connector and the output topic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org