You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/10/27 17:28:14 UTC
[GitHub] [pulsar-client-node] dlecocq opened a new issue #132: `AlreadyClosed` exception when using `topicsPattern`
dlecocq opened a new issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132
We are unable to cleanly close our consumer when using `topicsPattern`:
```
2020-10-27 11:15:43.304 INFO [0x70000d6d5000] ConnectionPool:85 | Created connection for pulsar://pulsar.data-access-platform-portal.docker:6650
2020-10-27 11:15:43.307 INFO [0x70000ef61000] ClientConnection:343 | [192.168.99.1:58802 -> 192.168.99.100:6650] Connected to broker
Closing consumer...
(node:35996) UnhandledPromiseRejectionWarning: Error: Failed to close consumer: AlreadyClosed
(node:35996) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:35996) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
```
That's the result from running this code snippet (with the appropriate docker container running):
```
const Bluebird = require('bluebird')
const Pulsar = require('pulsar-client')
const doit = async () => {
const client = new Pulsar.Client({
webServiceUrl: 'http://pulsar.data-access-platform-portal.docker:8080',
serviceUrl: 'pulsar://pulsar.data-access-platform-portal.docker:6650'
})
const consumer = await client.subscribe({
topicsPattern: `non-persistent://public/default/topic`,
subscription: 'test-subscription',
subscriptionType: 'Shared',
subscriptionInitialPosition: 'Earliest'
})
// Rather than push messages to consume, we're just waiting for a second
await Bluebird.delay(1000)
console.log('Closing consumer...')
// We've also tried omitting this line, but it still produces errors
await consumer.close()
console.log('Closing client...')
await client.close()
}
doit()
```
We've tried this with node versions 10.16.3, 12.16.0, and 14.11.0 and pulsar 2.6.1.
I _suspect_ that the root cause is that it _appears_ that the consumer implementation in this library makes the assumption that there's only a single consumer - https://github.com/apache/pulsar-client-node/blob/f010fb2ed3c530af8006ba3d3a7b73c6b46f6507/src/Consumer.cc#L178 overwrites the wrapper's consumer with each invocation. While this wouldn't be a problem with a single topic, it seems that the last consumer "wins" in that it gets stitched up to be the consumer saved in the wrapper. However, there may be a race condition for whether the listener handler stitched up correctly from the config (https://github.com/apache/pulsar-client-node/blob/f010fb2ed3c530af8006ba3d3a7b73c6b46f6507/src/Consumer.cc#L184 but then the consumer config is subsequently deleted).
Looking at `ClientImpl` in the pulsar C++ code, it appears that it also takes responsibility for closing any open consumers and producers, so we tried also just relying on the client in order to close the consumers (since `ClientImpl` appears to correctly track all newly-created consumers), but that _also_ reproduces with `AlreadyClosed` getting thrown.
All that said, it is a little tricky to track all of it because of the number of levels of indirection - from the JS to the C++ extension to the C bindings which delegate out to the C++ client code. So we might be way off 😆
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura commented on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura commented on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- persistent `topicsPattern` works
- `topic` and `topics` work
- values of other members aren't related
- it is not related whether or not RE is valid.
- an equivalent C++ client code works
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `non-persistent` → `persistent`: works
- `topicsPattern` → `topic` or `topics`: works
- other options aren't related
- it is not related whether or not the RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: though `AlreadyClosed` doesn't occur while unsubscribing, it does while closing the client.
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `non-persistent` → `persistent`: works
- `topicsPattern` → `topic` or `topics`: work
- other options aren't related
- it is not related whether or not RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: doesn't work (already closed)
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `non-persistent` → `persistent`: works
- `topicsPattern` → `topic` or `topics`: works
- other options aren't related
- it is not related whether or not the RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: doesn't work (already closed)
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura commented on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura commented on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-730811695
`AlreadyClosed` occurs when there're no matching topics for `topicsPattern`.
in such case [`consumers_`](https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc#L382) become empty and `closeAsync` passes `ResultAlreadyClosed` to the callback.
it may be because with `topicsPattern` a consumer can't dynamically create topics (since there is no means to generate all the possible string that match a given regular expression).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-730811695
`AlreadyClosed` occurs when there're no matching topics for `topicsPattern`.
in such case [`consumers_`](https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc#L382) become empty and `closeAsync` passes `ResultAlreadyClosed` to the callback.
(the same thing is also happening with C++ client but it is not explicitly output.)
it may be because with `topicsPattern` a consumer doesn't dynamically create topics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-730811695
`AlreadyClosed` occurs when there're no matching topics for `topicsPattern`.
in such case [`consumers_`](https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc#L382) become empty and `closeAsync` passes `ResultAlreadyClosed` to the callback.
(the same thing is also happening with C++ client.)
it may be because with `topicsPattern` a consumer doesn't dynamically create topics (there is no means to generate all the possible strings that match a given regular expression).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
~~it seems that this exception doesn't occur with persistent `topicsPattern`. 🤔~~
this was wrong. what mattered was whether or not matching topics exist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
it seems that this exception doesn't occur with persistent `topicsPattern`. 🤔
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `non-persistent` → `persistent`: works
- `topicsPattern` → `topic` or `topics`: work
- other options aren't related
- it is not related whether or not RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: adoesn't work (already closed)
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- persistent `topicsPattern` works
- non-persistent `topic` and `topics` work
- values of other members aren't related
- it is not related whether or not RE is valid.
- an equivalent C++ client code works
- using `unsubscribe` instead of `close` also doesn't work
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `non-persistent` → `persistent`: works
- `topicsPattern` → `topic` or `topics`: works
- other options aren't related
- it is not related whether or not the RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: the same result (already closed)
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
~~it seems that this exception doesn't occur with persistent `topicsPattern`. 🤔~~
i apologize that this was irrelevant (what mattered was whether or not the topic exists).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- `topicsPattern` → `topic` or `topics`: works
- other options aren't related
- it is not related whether or not the RE is valid.
- an equivalent C++ client code works
- `consumer.close` → `consumer.unsubscribe`: though `AlreadyClosed` doesn't occur while unsubscribing, it does while closing the client.
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-721479575
so far i've checked that
- persistent `topicsPattern` works
- non-persistent `topic` and `topics` work
- values of other members aren't related
- it is not related whether or not RE is valid.
- an equivalent C++ client code works
- it shows "already closed" when we use `unsubscribe` instead of `close` as well
i'll continue investigation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
~~it seems that this exception doesn't occur with persistent `topicsPattern`. 🤔~~
this was wrong. what mattered waswhether or not matching topics exist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura commented on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura commented on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
it seems that this exception doesn't occur with persistent topics ... 🤔
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-730811695
`AlreadyClosed` occurs when there're no matching topics for `topicsPattern`.
in such case [`consumers_`](https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/lib/MultiTopicsConsumerImpl.cc#L382) become empty and `closeAsync` passes `ResultAlreadyClosed` to the callback.
(the same thing is also happening with C++ client but it is not explicitly output.)
it may be because with `topicsPattern` a consumer doesn't dynamically create topics (there is no means to generate all the possible strings that match a given regular expression).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
it seems that this exception doesn't occur with persistent topics. 🤔
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-718509336
it seems that this exception doesn't occur with persistent `topicsPattern`. 🤔
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-node] cimura edited a comment on issue #132: `AlreadyClosed` exception when using `topicsPattern`
Posted by GitBox <gi...@apache.org>.
cimura edited a comment on issue #132:
URL: https://github.com/apache/pulsar-client-node/issues/132#issuecomment-730811695
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org