You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/07/07 23:35:52 UTC
[GitHub] [pulsar] nlu90 opened a new pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
nlu90 opened a new pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251
### Motivation
The Debezium requires pulsar a service URL for history database usage.
In #11056 , the `service.url` field from `PulsarKafkaWorkerConfig` is no longer available. And the value is also deleted from multiple yaml config files in this [commit](https://github.com/apache/pulsar/pull/11056/commits/3ce24c9e38c9420d60bfaf66aca92b78234082dc). This causes the integration test for Debezium connector to fail.
Based on the Debezium [paradigm](https://debezium.io/documentation/reference/1.5/connectors/mysql.html#debezium-mysql-connector-database-history-configuration-properties), all configurations should be passed as strings. There's no easy way to inject a PulsarClient via configuration.
We need to ask user to provide the pulsar url explicitly and probably auth info also.
### Modifications
1. Make the `database.history.pulsar.service.url` field required
2. Add the config value back to example yaml files
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-878796558
@sijie
> How is that related to CI in a very bad state since the integration tests for debezium is not enabled?
Sorry, I wasn't clear.
The point is that currently Debezium related tests are not running on CI, and I meant that if CI is not running the tests this is kind of a "bad state" :-)
We cannot enable them (https://github.com/apache/pulsar/pull/11154) because currently they will fail due to the problem that this PR is trying to address.
I would feel more confident with this change and any other proposal if we could revert #11056, commit #11154 and then be back in resolving the Client Auth problem.
So my proposal is:
- revert #11056 -> this prepares the way for #11154
- commit #11154 -> now we have CI that validates new changes
- work on this fix or on #11293
Does it sound like a good plan ?
because without #11154 we cannot validate on CI any patch related to Debezium
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
nlu90 commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-876855178
> @nlu90
>
> > One last way is we add history database specific pulsar auth parameters
>
> adding all parameters is tedious (the way config definition works there needs definition of each parameter with bunch of metadata around).
> What if we could define one parameter, String, for serialized config json and then do something like
>
> ```
> String clientConfJson = pulsarClient.getConfig();
> ```
>
> to pass that to the PulsarDatabaseHistory that can use ClientBuilder.loadConf(..)
@dlg99 This sounds an interesting idea!
I just checked the PulsarClient API, it doesn't support `getConfig` currently. We may want to discuss with the community about adding this API first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-876831351
@nlu90
> One last way is we add history database specific pulsar auth parameters
adding all parameters is tedious (the way config definition works there needs definition of each parameter with bunch of metadata around).
What if we could define one parameter, String, for serialized config json and then do something like
```
String clientConfJson = pulsarClient.getConfig();
```
to pass that to the PulsarDatabaseHistory that can use ClientBuilder.loadConf(..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-877202937
if this patch is unblocking @dlg99 's work to add back support for running Integration Tests of Pulsar IO
I would prefer to commit the patch in the current form and then follow up.
CI is in a very bad state currently.
cc @sijie @rdhabalia @merlimat @codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
nlu90 commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-876750952
> LGTM is this unblocks Debezium tests
>
> IIUC we wanted to use the client in order to access auth and TLS.
>
> With this change it is not possible to do it.
>
> What is your plan?
Based on Debezium's current paradigm, user can only pass string parameters in the configuration. For our case, it'll be better if they allow passing serialized objects. This will be the most secure way but needs help from Debezium community.
A second way would be we can access the PulsarClient via some static method in `PulsarDatabaseHistory`. This is a little bit hacky but works quickly if it's possible.
One last way is we add history database specific pulsar auth parameters in [`PulsarDatabaseHistory`](https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java#L53). This makes it follow the Debezium paradigm but not secure with pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-878771563
@nlu90 an alternative approach is to expose the client builder via the context. So we can just serialize the client builder to a base64 encoded string and pass it to debezium config. Hence the pulsar database history can get the client builder and create a Pulsar client from that. I created a PR (https://github.com/apache/pulsar/pull/11293) to demonstrate the idea. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-879188175
@sijie I second @eolivelli here.
We should either merge this change + fix for the CI or revert the breaking change, merge the fix for the CI, and do a follow up for the authentication after that.
At least the follow up change will get tested this time.
I don't find attractive idea of piling up more changes on top of this commit because it:
- has unpredictable ETA. Whatever way you decide to go with authentication will require separate discussion/CR, maybe even PIP.
- is blocking other changes (i.e. debezium upgrade)
- is leaving the debezium connectors in broken state (affects people testing/experimenting with pulsar on 2.9-snapshot)
- assumes that more changes added with the integration tests not enabled on the CI
We are dealing with a case of broken functionality in the build. I think the first priority should be fixing the regression quickly and new features should be second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#discussion_r666375364
##########
File path: tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMongoDbSourceTester.java
##########
@@ -50,7 +50,7 @@ public DebeziumMongoDbSourceTester(PulsarCluster cluster) {
sourceConfig.put("mongodb.password", "dbz");
sourceConfig.put("mongodb.task.id","1");
sourceConfig.put("database.whitelist", "inventory");
- sourceConfig.put("database.history.pulsar.service.url", pulsarServiceUrl);
+ sourceConfig.put("history.database.pulsar.service.url", pulsarServiceUrl);
Review comment:
I thought it should start with `DatabaseHistory.CONFIGURATION_FIELD_PREFIX_STRING` which is `CONFIGURATION_FIELD_PREFIX_STRING = "database.history."`
debezium does
```
Configuration dbHistoryConfig = config.subset(DatabaseHistory.CONFIGURATION_FIELD_PREFIX_STRING, false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-876901519
> A second way would be we can access the PulsarClient via some static method in PulsarDatabaseHistory. This is a little bit hacky but works quickly if it's possible.
@nlu90 I felt this is the most secure solution because the connector users don't need to provide the auth information through function config. The current PR still requires users to submit auth information via function config. Although I admit the code will look a bit hacky, it is still manageable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nlu90 commented on a change in pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
nlu90 commented on a change in pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#discussion_r666516477
##########
File path: tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMongoDbSourceTester.java
##########
@@ -50,7 +50,7 @@ public DebeziumMongoDbSourceTester(PulsarCluster cluster) {
sourceConfig.put("mongodb.password", "dbz");
sourceConfig.put("mongodb.task.id","1");
sourceConfig.put("database.whitelist", "inventory");
- sourceConfig.put("database.history.pulsar.service.url", pulsarServiceUrl);
+ sourceConfig.put("history.database.pulsar.service.url", pulsarServiceUrl);
Review comment:
Yeah, I just updated them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dlg99 commented on a change in pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
dlg99 commented on a change in pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#discussion_r666375364
##########
File path: tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMongoDbSourceTester.java
##########
@@ -50,7 +50,7 @@ public DebeziumMongoDbSourceTester(PulsarCluster cluster) {
sourceConfig.put("mongodb.password", "dbz");
sourceConfig.put("mongodb.task.id","1");
sourceConfig.put("database.whitelist", "inventory");
- sourceConfig.put("database.history.pulsar.service.url", pulsarServiceUrl);
+ sourceConfig.put("history.database.pulsar.service.url", pulsarServiceUrl);
Review comment:
I thought its should start with `DatabaseHistory.CONFIGURATION_FIELD_PREFIX_STRING` which is `CONFIGURATION_FIELD_PREFIX_STRING = "database.history."`
debezium does
```
Configuration dbHistoryConfig = config.subset(DatabaseHistory.CONFIGURATION_FIELD_PREFIX_STRING, false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #11251: [pulsar-io] pass the pulsar service url to debezium source for history database
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #11251:
URL: https://github.com/apache/pulsar/pull/11251#issuecomment-878739039
@eolivelli How is that related to CI in a very bad state since the integration tests for debezium is not enabled?
I would have a concern about this patch as it doesn't support accessing an authenticated Pulsar cluster yet. Even we merge this pull request for now, the connector can't be used in an authenticated cluster. I think we need to spend more time figuring out how to fix this properly. All the changes related are done in the master which is the development branch for 2.9 (which is targeted to release in September based on a time-based release schedule). I don't see why we need to rush on merging this. We need to fix this properly in order to make the debezium connector work in an authenticated cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org