[GitHub] merlimat opened a new issue #1143: Errors in increasing number of partitions for a partitioned topic

[GitBox <gi...@apache.org>]

merlimat opened a new issue #1143: Errors in increasing number of partitions for a partitioned topic
URL: https://github.com/apache/incubator-pulsar/issues/1143
 
 
   There are few errors that are happening under certain conditions when increasing the number of partitions for a topic.
   
   1. Increasing the partitions for a topic that was not used results in a 500 error with exception in the broker logs: 
   
   ```bash
   bin/pulsar-admin persistent create-partitioned-topic $TOPIC -p 10
   # and immediately after :
   bin/pulsar-admin persistent update-partitioned-topic $TOPIC -p 20
   ```
   
   ```
   2018-01-30 00:53:39,353 - WARN  - [bookkeeper-ml-workers-51-1:PersistentTopics$3@1506] - [null] Failed to get list of cursors of test/local/ns/persistent/my-topic-partition-1
   2018-01-30 00:53:39,353 - ERROR - [pulsar-web-69-23:PersistentTopics@463] - [null] Failed to update partitioned topic persistent://test/local/ns/my-topic
    org.apache.bookkeeper.mledger.ManagedLedgerException$MetaStoreException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
    Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper.lambda$null$5(MetaStoreImplZookeeper.java:202)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:30)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:31)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: at java.lang.Thread.run(Thread.java:748)
   Jan 30 00:53:39 ip-10-0-0-96.us-west-2.compute.internal pulsar[12513]: 2018-01-30 00:53:39,358 - INFO  - [pulsar-web-69-23:Slf4jRequestLog@60] - 10.0.0.100 - - [30/Jan/2018:00:53:39 +0000] "POST //10.0.0.96:8080/admin/persistent/test/local/ns/my-topic/partitions HTTP/1.1" 500 1045 "-" "Jersey/2.23.2 (HttpU
   ```
   
   This is due to the broker trying to get the list of existing subscriptions for the topic (with the purpose of pre-creating the subscriptions on the new partitions) and failing at it.
   
    2. In some condition the update also fails when the partitioned topic is fully created, because the HTTP request is not routed to the broker owning the particular partition (again when trying to pre-create the subscription).
   
   ```
   2018-01-30 00:58:48,482 - WARN  - [bookkeeper-ml-workers-44-1:BrokerService@867] - Namespace not served by this instance. Please redo the lookup. Request is denied: namespace=test/local/ns
   2018-01-30 00:58:48,482 - WARN  - [bookkeeper-ml-workers-44-1:BrokerService@439] - [persistent://test/local/ns/my-topic-partition-10] Service unit is not ready when loading the topic
   ``` 
   
   In this case the broker fails to validate the ownership and the REST request stays stuck with no response sent back to caller.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services