You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/21 14:32:53 UTC
[GitHub] [pulsar] frankjkelly opened a new issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
frankjkelly opened a new issue #9020:
URL: https://github.com/apache/pulsar/issues/9020
**Describe the bug**
Topic termination documentation says
```
Terminate a topic (disallow further messages from being published on the topic)
```
But once you terminate a topic, even if the retention settings allow data retention, a new consumer cannot read from it.
**To Reproduce**
Steps to reproduce the behavior:
1. Produce data to a topic
```
./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/frankTest7 -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
2. Terminate The topic
```
./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics terminate persistent://cogito-dialog/wav/frankTest7
```
3. Consume from the Topic
```
./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -s tcpingress-test2 -p Earliest persistent://cogito-dialog/wav/frankTest7 -t Shared
```
4. Get a message indicating we reached the end of the topic
```
09:12:07.353 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/frankTest7][tcpingress-test2] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/100.26.7.97:7443 -- consumer: 0
09:12:07.398 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/100.26.7.97:7443] Broker notification reached the end of topic: 0
09:12:07.399 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [tcpingress-test2] [persistent://cogito-dialog/wav/frankTest7] [8778c] Consumer has reached the end of topic
```
Retention period is 3 days
```
./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 namespaces get-retention cogito-dialog/wav
{
"retentionTimeInMinutes" : 4320,
"retentionSizeInMB" : -1
}
```
**Expected behavior**
The Admin Docs say
```Terminate a topic (disallow further messages from being published on the topic)```
https://pulsar.apache.org/docs/en/pulsar-admin/#terminate
Also the Javadoc says
```Terminate a topic. A topic that is terminated will not accept any more messages to be published and will let consumer to drain existing messages in backlog```
https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java#L2168-L2169
Neither says anything about future consumers
So either
1) Topic termination should still allow new consumers to read data from the topic
2) (If this works as designed) the Documentation should be updated to clearly indicate that
i) Existing consumers once they catch up to the end will be notified there is no more data
ii) New consumers will get NO data
**Additional context**
- Using Apache Pulsar 2.6.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-756553039
@frankjkelly
I have already tested our commands. I can't reproduce your behavior. I think you already have a subscription and the subscription already reach to end of the topic. When you try to consume from that, you will reach EndOfTopic because the subscription is already catchup and no messages to consume. You can reset the subscription to Earliest using `pulsar-admin reset-cursor`.
```
[sijie@Sijies-MacBook-Pro apache-pulsar-2.7.0]$ bin/pulsar-client produce -m "hello world" test-terminated
[sijie@Sijies-MacBook-Pro apache-pulsar-2.7.0]$ bin/pulsar-admin topics terminate test-terminated
Topic succesfully terminated at 11:0:-1
[sijie@Sijies-MacBook-Pro apache-pulsar-2.7.0]$ bin/pulsar-client consume -s new-sub -p Earliest test-terminated
21:18:48.931 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xbb326a22, L:/127.0.0.1:62666 - R:localhost/127.0.0.1:6650]] Connected to server
21:18:49.042 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {
"topicNames" : [ "test-terminated" ],
"topicsPattern" : null,
"subscriptionName" : "new-sub",
"subscriptionType" : "Exclusive",
"subscriptionMode" : "Durable",
"receiverQueueSize" : 1000,
"acknowledgementsGroupTimeMicros" : 100000,
"negativeAckRedeliveryDelayMicros" : 60000000,
"maxTotalReceiverQueueSizeAcrossPartitions" : 50000,
"consumerName" : null,
"ackTimeoutMillis" : 0,
"tickDurationMillis" : 1000,
"priorityLevel" : 0,
"maxPendingChuckedMessage" : 10,
"autoAckOldestChunkedMessageOnQueueFull" : false,
"expireTimeOfIncompleteChunkedMessageMillis" : 60000,
"cryptoFailureAction" : "FAIL",
"properties" : { },
"readCompacted" : false,
"subscriptionInitialPosition" : "Earliest",
"patternAutoDiscoveryPeriod" : 60,
"regexSubscriptionMode" : "PersistentOnly",
"deadLetterPolicy" : null,
"retryEnable" : false,
"autoUpdatePartitions" : true,
"autoUpdatePartitionsIntervalSeconds" : 60,
"replicateSubscriptionState" : false,
"resetIncludeHead" : false,
"keySharedPolicy" : null,
"batchIndexAckEnabled" : false
}
21:18:49.046 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650/",
"authPluginClassName" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxLookupRedirects" : 20,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"initialBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 60000000000,
"listenerName" : null,
"useKeyStoreTls" : false,
"sslProvider" : null,
"tlsTrustStoreType" : "JKS",
"tlsTrustStorePath" : "",
"tlsTrustStorePassword" : "",
"tlsCiphers" : [ ],
"tlsProtocols" : [ ],
"proxyServiceUrl" : null,
"proxyProtocol" : null,
"enableTransaction" : false
}
21:18:49.061 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [test-terminated][new-sub] Subscribing to topic on cnx [id: 0xbb326a22, L:/127.0.0.1:62666 - R:localhost/127.0.0.1:6650], consumerId 0
21:18:49.076 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [test-terminated][new-sub] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
21:18:49.091 [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
----- got message -----
key:[null], properties:[], content:hello world
21:18:49.109 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://localhost:6650/
21:18:49.122 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [localhost/127.0.0.1:6650] Broker notification reached the end of topic: 0
21:18:49.122 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [new-sub] [test-terminated] [15e27] Consumer has reached the end of topic
21:18:49.123 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [test-terminated] [new-sub] Closed consumer
21:18:49.125 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xbb326a22, L:/127.0.0.1:62666 ! R:localhost/127.0.0.1:6650] Disconnected
21:18:49.132 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully consumed
[sijie@Sijies-MacBook-Pro apache-pulsar-2.7.0]$ bin/pulsar-client produce -m "hello world 2" test-terminated
21:19:01.851 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa6301a34, L:/127.0.0.1:62669 - R:localhost/127.0.0.1:6650]] Connected to server
21:19:01.951 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
"topicName" : "test-terminated",
"producerName" : null,
"sendTimeoutMs" : 30000,
"blockIfQueueFull" : false,
"maxPendingMessages" : 1000,
"maxPendingMessagesAcrossPartitions" : 50000,
"messageRoutingMode" : "RoundRobinPartition",
"hashingScheme" : "JavaStringHash",
"cryptoFailureAction" : "FAIL",
"batchingMaxPublishDelayMicros" : 1000,
"batchingPartitionSwitchFrequencyByPublishDelay" : 10,
"batchingMaxMessages" : 1000,
"batchingMaxBytes" : 131072,
"batchingEnabled" : true,
"chunkingEnabled" : false,
"compressionType" : "NONE",
"initialSequenceId" : null,
"autoUpdatePartitions" : true,
"autoUpdatePartitionsIntervalSeconds" : 60,
"multiSchema" : true,
"properties" : { }
}
21:19:01.956 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650/",
"authPluginClassName" : null,
"operationTimeoutMs" : 30000,
"statsIntervalSeconds" : 60,
"numIoThreads" : 1,
"numListenerThreads" : 1,
"connectionsPerBroker" : 1,
"useTcpNoDelay" : true,
"useTls" : false,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"tlsHostnameVerificationEnable" : false,
"concurrentLookupRequest" : 5000,
"maxLookupRequest" : 50000,
"maxLookupRedirects" : 20,
"maxNumberOfRejectedRequestPerConnection" : 50,
"keepAliveIntervalSeconds" : 30,
"connectionTimeoutMs" : 10000,
"requestTimeoutMs" : 60000,
"initialBackoffIntervalNanos" : 100000000,
"maxBackoffIntervalNanos" : 60000000000,
"listenerName" : null,
"useKeyStoreTls" : false,
"sslProvider" : null,
"tlsTrustStoreType" : "JKS",
"tlsTrustStorePath" : "",
"tlsTrustStorePassword" : "",
"tlsCiphers" : [ ],
"tlsProtocols" : [ ],
"proxyServiceUrl" : null,
"proxyProtocol" : null,
"enableTransaction" : false
}
21:19:01.965 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [test-terminated] [null] Creating producer on cnx [id: 0xa6301a34, L:/127.0.0.1:62669 - R:localhost/127.0.0.1:6650]
21:19:01.974 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xa6301a34, L:/127.0.0.1:62669 - R:localhost/127.0.0.1:6650] Received error from server: Topic was already terminated
21:19:01.975 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [test-terminated] [null] Failed to create producer: Topic was already terminated
21:19:01.976 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
21:19:01.976 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Topic was already terminated
org.apache.pulsar.client.api.PulsarClientException$TopicTerminatedException: Topic was already terminated
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:932) ~[org.apache.pulsar-pulsar-client-api-2.7.0.jar:2.7.0]
at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93) ~[org.apache.pulsar-pulsar-client-original-2.7.0.jar:2.7.0]
at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:211) [org.apache.pulsar-pulsar-client-tools-2.7.0.jar:2.7.0]
at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:196) [org.apache.pulsar-pulsar-client-tools-2.7.0.jar:2.7.0]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:175) [org.apache.pulsar-pulsar-client-tools-2.7.0.jar:2.7.0]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:209) [org.apache.pulsar-pulsar-client-tools-2.7.0.jar:2.7.0]
21:19:01.979 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-763178272
I retried with the command line tools for Pulsar 2.6.1 ensuring that the subscriber name was new each time and was able to reproduce the behavior
```
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriber5] [persistent://cogito-dialog/wav/test5] [48326] Consumer has reached the end of topic
```
**TEST CASE 1: No Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testNoTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberNoTerminate persistent://cogito-dialog/wav/testNoTerminate -t Shared
```
Response
```
17:15:31.672 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribing to topic on cnx [id: 0x4a36f046, L:/10.250.8.90:62168 - R:pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443]
17:15:31.725 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443 -- consumer: 0
```
**TEST CASE 2: Include Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testWithTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Termination
```
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics terminate persistent://cogito-dialog/wav/testWithTerminate
Warning: Nashorn engine is planned to be removed from a future JDK release
Topic succesfully terminated at 951286:0:-1
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberWithTerminate persistent://cogito-dialog/wav/testWithTerminate -t Shared
```
Response
```
17:17:42.151 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribing to topic on cnx [id: 0x2e169889, L:/10.250.8.90:62185 - R:pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443]
17:17:42.293 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443 -- consumer: 0
17:17:42.477 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:17:42.478 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriberWithTerminate] [persistent://cogito-dialog/wav/testWithTerminate] [be3a5] Consumer has reached the end of topic
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-756550606
@frankjkelly Does the subscription already exist?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-751207139
@merlimat @sijie I'm not sure how to deal with consumers at the beginning of the design, in my opinion, if the topic terminated, the broker should prevent all the producers and consumers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
frankjkelly edited a comment on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-763178272
I retried with the command line tools for Pulsar 2.6.1 ensuring that the subscriber name was new each time and was able to reproduce the behavior
```
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriber5] [persistent://cogito-dialog/wav/test5] [48326] Consumer has reached the end of topic
```
**TEST CASE 1: No Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testNoTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberNoTerminate persistent://cogito-dialog/wav/testNoTerminate -t Shared
```
Response
```
17:15:31.672 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribing to topic on cnx [id: 0x4a36f046, L:/10.250.8.90:62168 - R:pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443]
17:15:31.725 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443 -- consumer: 0
```
**TEST CASE 2: Include Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testWithTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Termination
```
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics terminate persistent://cogito-dialog/wav/testWithTerminate
Warning: Nashorn engine is planned to be removed from a future JDK release
Topic succesfully terminated at 951286:0:-1
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberWithTerminate persistent://cogito-dialog/wav/testWithTerminate -t Shared
```
Response
```
17:17:42.151 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribing to topic on cnx [id: 0x2e169889, L:/10.250.8.90:62185 - R:pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443]
17:17:42.293 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443 -- consumer: 0
17:17:42.477 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:17:42.478 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriberWithTerminate] [persistent://cogito-dialog/wav/testWithTerminate] [be3a5] Consumer has reached the end of topic
```
NOTE: If I do the reset-cursor
```
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics reset-cursor persistent://cogito-dialog/wav/testWithTerminate -s subcriberWithTerminate -t 2d
```
I *do* get data back
But again it appears surprising that a new subscription, created *after* topic termination, needs to have its cursor reset
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-756550630
I think there might be additional nuance to this situation, specifically regarding subscriptions and consumers. In @frankjkelly's example, the subscription is created after the topic is deleted. I'm pretty sure an existing subscription would allow for a consumer to connect to it because the subscription owns the cursor, which would already be set. I wasn't able to test out this theory, but if it is correct, I think the question here becomes whether or not users can create subscriptions for deleted topics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly edited a comment on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
frankjkelly edited a comment on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-763178272
I retried with the command line tools for Pulsar 2.6.1 ensuring that the subscriber name was new each time and was able to reproduce the behavior
```
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:13:18.081 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriber5] [persistent://cogito-dialog/wav/test5] [48326] Consumer has reached the end of topic
```
**TEST CASE 1: No Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testNoTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberNoTerminate persistent://cogito-dialog/wav/testNoTerminate -t Shared
```
Response
```
17:15:31.672 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribing to topic on cnx [id: 0x4a36f046, L:/10.250.8.90:62168 - R:pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443]
17:15:31.725 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testNoTerminate][subcriberNoTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/3.217.29.231:7443 -- consumer: 0
```
**TEST CASE 2: Include Terminate**
Producer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI produce persistent://cogito-dialog/wav/testWithTerminate -f ~/platform2-signal/signal-stream-client-library/src/intTest/resources/python.wav
```
Termination
```
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics terminate persistent://cogito-dialog/wav/testWithTerminate
Warning: Nashorn engine is planned to be removed from a future JDK release
Topic succesfully terminated at 951286:0:-1
```
Consumer
```
$ ./pulsar-client --url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-url pulsar+ssl://pulsar.saas-dev.cogitocorp.us:7443 --proxy-protocol SNI consume -p Earliest -s subcriberWithTerminate persistent://cogito-dialog/wav/testWithTerminate -t Shared
```
Response
```
17:17:42.151 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribing to topic on cnx [id: 0x2e169889, L:/10.250.8.90:62185 - R:pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443]
17:17:42.293 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://cogito-dialog/wav/testWithTerminate][subcriberWithTerminate] Subscribed to topic on pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443 -- consumer: 0
17:17:42.477 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [pulsar.saas-dev.cogitocorp.us/54.82.124.115:7443] Broker notification reached the end of topic: 0
17:17:42.478 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [subcriberWithTerminate] [persistent://cogito-dialog/wav/testWithTerminate] [be3a5] Consumer has reached the end of topic
```
NOTE: If I do the reset-cursor
```
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 topics reset-cursor persistent://cogito-dialog/wav/testWithTerminate -s subcriberWithTerminate -t 2d
```
I *do* get data back
But again it seems *surprising* that a new subscription, created after topic termination, needs to have its cursor reset
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] frankjkelly commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
frankjkelly commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-753997738
Hi @codelipenghui @merlimat @sijie any thoughts or updates on this? Thanks in advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall edited a comment on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
michaeljmarshall edited a comment on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-756550630
I think there might be additional nuance to this situation, specifically regarding subscriptions and consumers. In @frankjkelly's example, the subscription is created after the topic is deleted. I'm pretty sure an existing subscription would allow for a consumer to connect to it because the subscription owns the cursor, which would already be set. I wasn't able to test out this theory, but if it is correct, I think the question here becomes whether or not users can create subscriptions for deleted topics.
Edit: I'm assuming auto subscription creation is turned on here, and that the subscription was not already created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9020: Once a topic is terminated it can no longer be read from by a new Consumer
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9020:
URL: https://github.com/apache/pulsar/issues/9020#issuecomment-1058894503
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org