You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/04/29 16:24:57 UTC
[GitHub] [pulsar] itaross opened a new issue #6845: Subscription to topics with regex not working for non-persisten topics
itaross opened a new issue #6845:
URL: https://github.com/apache/pulsar/issues/6845
Messages on non-persistent topics are not received when using a regex subscription.
This problem is encountered when using a Pulsar IO custom connector (sink).
When using persistent topics the subscription is working as expected
**Expected behavior**
Records are received when subscribing to regex topics.
**Additional context**
Issues #2009
https://github.com/apache/pulsar/issues/2009
Issues #2025
https://github.com/apache/pulsar/issues/2025
Does not solve this problem
@merlimat @sijie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] itaross commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
itaross commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-621711182
I am currently using version 2.5.1
I think you can reproduce it even with a builting connector.
Create it from the coommand line, using --topics-pattern and use a non-persistent:// based pattern.
You will see that no data is received in the write method.
This has been highlighted also in the previous cited issued, but it has not been addressed in sinks. Everything works if you you use persistent topics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] itaross commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
itaross commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-629610766
Hi @wolfstudy, any update on this? May I help with this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nicolo-paganin commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
nicolo-paganin commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-624706973
@codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] itaross commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
itaross commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-625288758
> > @codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful
>
> Can you provide a way to reproduce? In functions, sinks and sources, `--topics-pattern` will be deprecated, can you use `--inputs` replace it? [--input] and [--topic-pattern] are mutually exclusive.
@wolfstudy what you are referring to is probably this line https://github.com/oncodeit/pulsar/blob/71b3819bfc11afe0f0da19df0a19e3b7a05988c5/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java#L258
but this involves only the name of the pattern parameter
To reproduce: take any connector from Pulsar IO, a sink in this case.
https://archive.apache.org/dist/pulsar/pulsar-2.5.1/connectors/
Follow these steps: https://pulsar.apache.org/docs/en/standalone/#install-builtin-connectors-optional
Launch the connector with:
```
./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc --sink-config-file <the yaml config file> --topicsPattern "persistent://public/default/.*"
```
content of the yaml:
````
{
"userName": "root",
"password": "jdbc",
"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
"tableName": "pulsar_mysql_jdbc_sink"
}
```
then retry using:
```
./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc --sink-config-file <the yaml config file> --topicsPattern "non-persistent://public/default/.*"
```
You will notice that the connector will now works.
I posted this example with jdbc/mysql but you can try with anything you prefer, the behaviour is the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy edited a comment on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
wolfstudy edited a comment on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-625236256
> @codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful
Can you provide a way to reproduce? In functions, sinks and sources, `--topics-pattern` will be deprecated, can you use `--inputs` replace it? [--input] and [--topic-pattern] are mutually exclusive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] wolfstudy commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-625236256
> @codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful
Can you provide a way to reproduce? In functions, sinks and sources, `--topics-pattern` will be deprecated, can you use `--inputs` replace it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-621642156
@itaross Can you provide more details about the bug? what version are you using and how to reproduce it. This will help to troubleshoot this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] itaross edited a comment on issue #6845: Subscription to topics with regex not working for non-persisten topics
Posted by GitBox <gi...@apache.org>.
itaross edited a comment on issue #6845:
URL: https://github.com/apache/pulsar/issues/6845#issuecomment-625288758
> > @codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful
>
> Can you provide a way to reproduce? In functions, sinks and sources, `--topics-pattern` will be deprecated, can you use `--inputs` replace it? [--input] and [--topic-pattern] are mutually exclusive.
@wolfstudy what you are referring to is probably this line https://github.com/oncodeit/pulsar/blob/71b3819bfc11afe0f0da19df0a19e3b7a05988c5/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java#L258
but this involves only the name of the pattern parameter
To reproduce: take any connector from Pulsar IO, a sink in this case.
https://archive.apache.org/dist/pulsar/pulsar-2.5.1/connectors/
Follow these steps: https://pulsar.apache.org/docs/en/standalone/#install-builtin-connectors-optional
Launch the connector with:
```
./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc --sink-config-file <the yaml config file> --topicsPattern "persistent://public/default/.*"
```
content of the yaml:
```
{
"userName": "root",
"password": "jdbc",
"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
"tableName": "pulsar_mysql_jdbc_sink"
}
```
then retry using:
```
./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc --sink-config-file <the yaml config file> --topicsPattern "non-persistent://public/default/.*"
```
You will notice that the connector will now works.
I posted this example with jdbc/mysql but you can try with anything you prefer, the behaviour is the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org