You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2018/12/28 20:02:17 UTC
[GitHub] HaraldGustafsson opened a new issue #2952: Setting topic permission
under load
HaraldGustafsson opened a new issue #2952: Setting topic permission under load
URL: https://github.com/apache/pulsar/issues/2952
#### Expected behavior
Do a REST call to admin interface for setting topic permissions like:
https://localhost:8081/admin/v2/persistent/tt1/testns1/t4%5Cdata/permissions/testrole
body:
{'testrole': ['consume', 'produce']}
Should work every time and return status 204.
#### Actual behavior
When this is done with heavy load setting up permission for many topics in parallel. Some will result in the below logged exception and status code 500.
#### Steps to reproduce
Set up tenant and namespace, then run the set permission for many different topics in that namespace in parallel. I use the same role and actions for all of them.
#### System configuration
pulsar: 2.2.0
This error was when running the broker in standalone mode in a docker container supplied on docker hub, with authorisation and authentication via client certs. Have not tested on our full pulsar setup yet.
#### Logs
12:27:31.613 [pulsar-web-53-28] INFO org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superUser] Successfully granted access for role testrole: [produce, consume] - topic persistent://tt1/testns1/t5\meta
12:27:31.587 [pulsar-web-53-30] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [superUser] Failed to grant permissions for topic persistent://tt1/testns1/t0\data
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /admin/policies/tt1/testns1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1336) ~[org.apache.pulsar-pulsar-zookeeper-2.2.0.jar:2.2.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$3101(ZooKeeperClient.java:70) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$21.call(ZooKeeperClient.java:1065) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$21.call(ZooKeeperClient.java:1059) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient.setData(ZooKeeperClient.java:1059) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGrantPermissionsOnTopic(PersistentTopicsBase.java:274) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.apache.pulsar.broker.admin.v2.PersistentTopics.grantPermissionsOnTopic(PersistentTopics.java:107) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [org.glassfish.jersey.core-jersey-common-2.25.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [org.glassfish.jersey.core-jersey-server-2.25.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.25.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) [org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:74) [org.apache.pulsar-pulsar-broker-common-2.2.0.jar:2.2.0]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [org.eclipse.jetty-jetty-servlet-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.Server.handle(Server.java:524) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [org.eclipse.jetty-jetty-server-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [org.eclipse.jetty-jetty-io-9.3.11.v20160721.jar:9.3.11.v20160721]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [org.apache.pulsar-pulsar-functions-metrics-2.2.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
12:27:31.669 [pulsar-web-53-27] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t6%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 385
12:27:31.669 [pulsar-web-53-31] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t7%5Cmeta/permissions/testrole HTTP/1.1" 500 5652 "-" "Python/3.7 aiohttp/3.4.2" 402
12:27:31.668 [pulsar-web-53-29] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t8%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 371
12:27:31.643 [pulsar-web-53-28] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t5%5Cmeta/permissions/testrole HTTP/1.1" 204 0 "-" "Python/3.7 aiohttp/3.4.2" 173
12:27:31.642 [pulsar-web-53-24] INFO org.eclipse.jetty.server.RequestLog - 172.17.0.1 - - [07/Nov/2018:12:27:31 +0000] "POST https://localhost:8081/admin/v2/persistent/tt1/testns1/t4%5Cdata/permissions/testrole HTTP/1.1" 500 6238 "-" "Python/3.7 aiohttp/3.4.2" 341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services