You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/01/11 12:16:48 UTC
[GitHub] [pulsar] codelipenghui opened a new issue #9173: Create partitions even if topic auto-creation enabled
codelipenghui opened a new issue #9173:
URL: https://github.com/apache/pulsar/issues/9173
**Describe the bug**
Currently, for the persistent topic. If create a new partitioned persistent topic, we also create partitions. So even if users disabled the topic auto-creation, the producers and consumers also can connect to the partitions. But for the non-persistent topic. It needs to enable the topic auto-creation, otherwise, the partitions will not be created since we don't maintain any partition metadata in the metadata service. See https://github.com/apache/pulsar/pull/9107
But, enabled the topic auto-creation will allow any new topics from the producers and consumers. From the Pulsar maintainer perspective, if the partitioned topic created, this means allows to create the partitons of the partitioned topic.
**Expected behavior**
Even if the topic auto-creation disabled, the partitions should be created automatically if the partitioned topic created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789476109
@codelipenghui What if the partitions are created automatically when the partitioned topic is created? Like for persistent topics ?
So we keep a consistent behavior on topics creation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789783373
@mlyahmed No, Pulsar does not store any metadata for the non-persistent topic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789987918
Here is a suggestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-862107364
It seems non-persistent part of this issue is fixed.
I am glad to fix the case of persistent topics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-788877352
I'll be happy to participate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789519168
@codelipenghui when the partitions, for non persistent topics, created are those metadata created?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789416805
@mlyahmed 2.7.0 does not contain https://github.com/apache/pulsar/pull/9029, so you can connect to the non-persistent topic partitions. You can try it based on the last master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789833502
@codelipenghui got that thanks !
This auto-creation must be only for partitions right? The current behavior must be kept for the non-partitioned topics. For example, if a new subscription to a non-persistent regular topic and it does not exist the topic must not be created. Right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-788533829
Move this to the next release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #9173:
URL: https://github.com/apache/pulsar/issues/9173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
Jason918 commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-862120596
@codelipenghui Here is how I reproduce this case for persistent topics.
1. set `allowAutoTopicCreation=false` in conf/standalone.conf
2. start pulsar standalone with `bin/pulsar standalone`
3. create partition topic with `bin/pulsar-admin topics create-partitioned-topic persistent://public/default/jason -p 4`
4. successfully produce one message to the topic with `bin/pulsar-client produce persistent://public/default/jason --messages "HELLO"`
5. list topics with `bin/pulsar-admin topics list public/default`. Here is results:
"persistent://public/default/jason-partition-1"
"persistent://public/default/jason-partition-0"
"persistent://public/default/jason-partition-3"
"persistent://public/default/jason-partition-2"
6. **Everything seems right for now.**
7. Produce one message to one of the partitions with `bin/pulsar-client produce persistent://public/default/jason-partition-0 --messages "HELLO"`. Failed with the following error: org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic Not Found.
8. Reproduce one message to the origin partitioned topic with `bin/pulsar-client produce persistent://public/default/jason --messages "HELLO"`. And **we got ERROR**, here is the main logs for client.
> 15:00:57.766 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-0] [null] Creating producer on cnx [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650]
> 15:00:57.771 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-1] [null] Creating producer on cnx [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650]
> 15:00:57.772 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-2] [null] Creating producer on cnx [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650]
> 15:00:57.772 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-3] [null] Creating producer on cnx [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650]
> 15:00:57.777 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650] Received error from server: Topic Not Found.
> 15:00:57.778 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-0] [null] Failed to create producer: Topic Not Found.
> 15:00:57.778 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-0] [null] Closed Producer (not connected)
> 15:00:57.784 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650] Received error from server: Topic Not Found.
> 15:00:57.784 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-1] [null] Failed to create producer: Topic Not Found.
> 15:00:57.784 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-1] [null] Closed Producer (not connected)
> 15:00:57.789 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650] Received error from server: Topic Not Found.
> 15:00:57.790 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-2] [null] Failed to create producer: Topic Not Found.
> 15:00:57.790 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-2] [null] Closed Producer (not connected)
> 15:00:57.798 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xb84eef3e, L:/127.0.0.1:59022 - R:localhost/127.0.0.1:6650] Received error from server: Topic Not Found.
> 15:00:57.798 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-3] [null] Failed to create producer: Topic Not Found.
> 15:00:57.798 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/jason-partition-3] [null] Closed Producer (not connected)
> 15:00:57.798 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.PartitionedProducerImpl - [persistent://public/default/jason] Could not create partitioned producer.
> 15:00:57.799 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.PartitionedProducerImpl - [persistent://public/default/jason] Closed Partitioned Producer
> 15:00:57.799 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages
> 15:00:57.799 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Topic Not Found.
> org.apache.pulsar.client.api.PulsarClientException$TopicDoesNotExistException: Topic Not Found.
> at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1019) ~[pulsar-client-api.jar:2.9.0-SNAPSHOT]
> at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:95) ~[pulsar-client-original.jar:2.9.0-SNAPSHOT]
> at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:223) [pulsar-client-tools.jar:2.9.0-SNAPSHOT]
> at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:204) [pulsar-client-tools.jar:2.9.0-SNAPSHOT]
> at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:186) [pulsar-client-tools.jar:2.9.0-SNAPSHOT]
> at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:222) [pulsar-client-tools.jar:2.9.0-SNAPSHOT]
> 15:00:57.803 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced
>
Here is the main error log in server:
> 15:20:29.192 [pulsar-io-29-8] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /127.0.0.1:59462
> 15:20:29.318 [pulsar-io-29-8] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:59462][persistent://public/default/jason-partition-0] Creating producer. producerId=0
> 15:20:29.319 [pulsar-ordered-OrderedExecutor-2-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/default/persistent/jason-partition-0
> 15:20:29.320 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/jason-partition-0] Closing managed ledger
> 15:20:29.320 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] ERROR org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:59462] Failed to create topic persistent://public/default/jason-partition-0, producerId=0
> java.util.concurrent.CompletionException: java.util.NoSuchElementException: No value present
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:604) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_91]
> at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:1261) ~[pulsar-broker.jar:2.9.0-SNAPSHOT]
> at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:402) ~[managed-ledger.jar:2.9.0-SNAPSHOT]
> at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_91]
> at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:397) ~[managed-ledger.jar:2.9.0-SNAPSHOT]
> at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$1.operationFailed(ManagedLedgerImpl.java:396) ~[managed-ledger.jar:2.9.0-SNAPSHOT]
> at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getManagedLedgerInfo$2(MetaStoreImpl.java:87) ~[managed-ledger.jar:2.9.0-SNAPSHOT]
> at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) [?:1.8.0_91]
> at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) [?:1.8.0_91]
> at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) [?:1.8.0_91]
> at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) [bookkeeper-common-4.14.1.jar:4.14.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_91]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.63.Final.jar:4.1.63.Final]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.util.NoSuchElementException: No value present
> at java.util.Optional.get(Optional.java:135) ~[?:1.8.0_91]
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_91]
> ... 24 more
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789016596
@codelipenghui can you please share more details on the way you proceed to reproduce the issue ?
Here is how I've tested and the results.
### Auto Creation **Enabled**
By keeping the auto-creation enabled, `allowAutoTopicCreation=true`, I created a **non-persistent partitioned topic**:
` bin/pulsar-admin topics create-partitioned-topic non-persistent://public/default/issue-9173 -p 3`
The partitioned topic is created **but not the partitions**.
When I create a new subscription or a producer the partitions are created **but also a regular topic with the same as the partitioned topic** :
After a certain time of inactivity, the regular topic, auto-created, is deleted.
The partitioned topic and its partitions are not deleted.
### Auto Creation **Disabled**
By disabling the auto-creation, `allowAutoTopicCreation=false`, I noticed the same behavior except that after a certain time of inactivity both the partitions and the regular topic, auto-created, are deleted. The partitioned topic is kept.
Hence the partitions are auto-created even when the auto-creation is disabled.
**The "odd" behavior that I noticed is the auto-creation of that regular topic with the same name as the partitioned topic.**
Pulsar version: 2.7.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789507324
@mlyahmed For the persistent topic, we can create all partitions when creating the partitioned topic, the metadata for each partition also maintains in the zookeeper. But for the non-persistent topic, we don't need to store any metadata for the partition, so we can't keep consistent with the current persistent topic behavior
I think we can fix the non-persistent topic case directly, if the non-persistent partitioned topic is created, we should allow creating partitions even if the topic auto-creation disabled so that the client able to connect to partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mlyahmed edited a comment on issue #9173: Create partitions for a partitioned topic even if topic auto-creation disabled
Posted by GitBox <gi...@apache.org>.
mlyahmed edited a comment on issue #9173:
URL: https://github.com/apache/pulsar/issues/9173#issuecomment-789476109
got that thanks. And I can reproduce it.
@codelipenghui What if the partitions are created automatically when a non-persistent partitioned topic is created? Like for persistent topics?
So we keep a consistent behavior on topics creation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org