You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/21 09:15:55 UTC
[GitHub] [pulsar] dragonls opened a new issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
dragonls opened a new issue #10300:
URL: https://github.com/apache/pulsar/issues/10300
**Describe the bug**
While using partitioned topic, if we control the authorization of the topic in topic level and update the topic partiton number, new producer/consumer will recerive auth error `org.apache.pulsar.client.api.PulsarClientException$LookupException: Proxy Client is not authorized to Lookup`
**To Reproduce**
Steps to reproduce the behavior:
1. Having a pulsar cluster using JWT for authentication
2. Create a partitionen topic `persistent://test/auth/topic` with 2 partitions: `./bin/pulsar-admin topics create-partitioned-topic -p 2 persistent://test/auth/topic`
3. Grant `produce` and `consume` permissions to role `roleA`: `./bin/pulsar-admin topics grant-permission --role roleA --actions produce,consume persistent://test/auth/topic`
4. Try produce and consume this topic using subscription `roleA-test`, shoule be normal till now
5. Update the topic partition number: `./bin/pulsar-admin topics update-partitioned-topic -p 4 persistent://test/auth/topic`
5. Try to use role `roleA` to produce/consume the topic using subscription `roleA-test` again, we will get error:
- produce get error: `org.apache.pulsar.client.api.PulsarClientException$LookupException: Proxy Client is not authorized to Lookup`
- consume get error: `org.apache.pulsar.client.api.PulsarClientException$LookupException: Failed to subscribe persistent://test/auth/topic with 4 partitions`
**Expected behavior**
After updating the partitioned topic, new producer/consumer of this topic should not get error.
**Additional context**
I tried branch 2.7 and master, both having this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] linlinnn edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
linlinnn edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-823957311
This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the new partition number of topic, but `roleA` do not have been granted `LOOKUP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824800393
There are 2 ways to solve this bug:
1. After updating the partition num of the topic, try to regrant the permissions of the topic. It is to say that in the end of [`org.apache.pulsar.broker.admin.impl.PersistentTopicsBase#updatePartitionedTopic`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java#L3446) , try to get the topic level permissions of the topic and regrant them by calling `internalGrantPermissionsOnTopic`.
2. In [`org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider#checkPermission`](https://github.com/apache/pulsar/blob/889b9b8e5efc62d2d0cbc761205fba5759c97af0/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L394) , for example, if the topic is a sub-partitioned topic `persistent://test/auth/topic-partition-4` of the partitioned topic `persistent://test/auth/topic`, try to check the permission of `persistent://test/auth/topic` again.
I can not tell which solution is better.
The first solution may change the permissions of previous sub-partitioned topics, and make `updatePartitionedTopic` heavy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824012171
> This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the new partition number of topic, but `Proxy Client` do not have been granted `LOOKUP`
Is that any solution or suggestion for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824012171
> This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the new partition number of topic, but `Proxy Client` do not have been granted `LOOKUP`
Is there any solution or suggestion for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824760511
> @dragonls you can add such config in broker.conf
>
> ```
> proxyRoles=my-proxy-role
> superUserRoles=my-proxy-role
> ```
@linlinnn I am using pulsar on k8s, and tried the config as you said. Still getting the same error.
We can look into the exception, https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L408 , it is not about pulsar proxy component but the authorization control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824760511
> @dragonls you can add such config in broker.conf
>
> ```
> proxyRoles=my-proxy-role
> superUserRoles=my-proxy-role
> ```
@linlinnn I am using pulsar on k8s, and tried the config as you said. Still getting the same error.
We can look into the exception, https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L408 , it is not about pulsar proxy component but the authorization control.
The detailed error log while using `./bin/pulsar-perf` to consume a patitioned topic, which is updated from 4 partitions to 5:
```
19:45:08.291 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-0][roleA-test] Subscribed to topic on pulsar-broker-13.pulsar-broker.pulsar.svc.cluster.local/9.165.129.10:6650 -- consumer: 0
19:45:08.291 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-1][roleA-test] Subscribed to topic on pulsar-broker-13.pulsar-broker.pulsar.svc.cluster.local/9.165.129.10:6650 -- consumer: 1
19:45:08.291 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-2][roleA-test] Subscribed to topic on pulsar-broker-13.pulsar-broker.pulsar.svc.cluster.local/9.165.129.10:6650 -- consumer: 2
19:45:08.293 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-3][roleA-test] Subscribed to topic on pulsar-broker-13.pulsar-broker.pulsar.svc.cluster.local/9.165.129.10:6650 -- consumer: 3
19:45:08.293 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - [persistent://test/auth/topic] Failed to subscribe for topic [persistent://test/auth/topic] in topics consumer org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Proxy Client is not authorized to Lookup
19:45:08.294 [pulsar-external-listener-3-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-4] [roleA-test] Closed Consumer (not connected)
19:45:08.298 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-3] [roleA-test] Closed consumer
19:45:08.299 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-2] [roleA-test] Closed consumer
19:45:08.299 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-1] [roleA-test] Closed consumer
19:45:08.300 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://test/auth/topic-partition-0] [roleA-test] Closed consumer
19:45:08.300 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - [persistent://test/auth/topic] Failed to subscribe for topic [persistent://test/auth/topic] in topics consumer, subscribe error: org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Proxy Client is not authorized to Lookup
19:45:08.300 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - Failed subscription for createPartitionedConsumer: persistent://test/auth/topic 5, e:java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Proxy Client is not authorized to Lookup
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Failed to subscribe persistent://test/auth/topic with 5 partitions
Proxy Client is not authorized to Lookup
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.pulsar.testclient.PerformanceConsumer.main(PerformanceConsumer.java:337)
Caused by: org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Failed to subscribe persistent://test/auth/topic with 5 partitions
Proxy Client is not authorized to Lookup
at org.apache.pulsar.client.api.PulsarClientException.wrap(PulsarClientException.java:845)
at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$createPartitionedConsumer$35(MultiTopicsConsumerImpl.java:857)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$null$42(MultiTopicsConsumerImpl.java:1015)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.pulsar.client.impl.ConsumerImpl.cleanupAtClose(ConsumerImpl.java:1079)
at org.apache.pulsar.client.impl.ConsumerImpl.lambda$closeAsync$15(ConsumerImpl.java:1064)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.pulsar.client.impl.ClientCnx.handleSuccess(ClientCnx.java:444)
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:268)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
```
The error log shows that not authorized to Lookup `persistent://test/auth/topic-partition-4`, which is the new partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824800393
There are 2 ways to solve this bug:
1. After updating the partition num of the topic, try to regrant the permissions of the topic. It is to say that in the end of [`org.apache.pulsar.broker.admin.impl.PersistentTopicsBase#updatePartitionedTopic`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java#L3446) , try to get the topic level permissions of the topic and regrant them by calling `internalGrantPermissionsOnTopic`.
2. In [`org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider#checkPermission`](https://github.com/apache/pulsar/blob/889b9b8e5efc62d2d0cbc761205fba5759c97af0/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L394) , for example, if the topic is a sub-partitioned topic `persistent://test/auth/topic-partition-4` of the partitioned topic `persistent://test/auth/topic`, try to check the permission of `persistent://test/auth/topic` again.
I can not tell which solution is better.
The first solution but that may change the permissions of previous sub-partitioned topics, and make `updatePartitionedTopic` heavy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] linlinnn commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
linlinnn commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824033219
@dragonls you can add such config in broker.conf
```
proxyRoles=my-proxy-role
superUserRoles=my-proxy-role
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] linlinnn commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
linlinnn commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-823957311
This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the partition number of topic, but `roleA` do not have been granted `LOOKUP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824800393
There are 2 ways to solve this bug:
1. After updating the partition num of the topic, try to regrant the permissions of the topic. It is to say that in the end of [`org.apache.pulsar.broker.admin.impl.PersistentTopicsBase#updatePartitionedTopic`](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java#L3446) , try to get the topic level permissions of the topic and regrant them by calling `internalGrantPermissionsOnTopic`.
2. In [`org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider#checkPermission`](https://github.com/apache/pulsar/blob/889b9b8e5efc62d2d0cbc761205fba5759c97af0/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authorization/PulsarAuthorizationProvider.java#L394) , for example, if the topic is a sub-partitioned topic `persistent://test/auth/topic-partition-4` of the partitioned topic `persistent://test/auth/topic`, try to check the permission of `persistent://test/auth/topic` again.
I can not tell which solution is better.
I prefer the first solution but that may change the permissions of previous sub-partitioned topics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] linlinnn edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
linlinnn edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-823957311
This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the new partition number of topic, but `Proxy Client` do not have been granted `LOOKUP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dragonls edited a comment on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
dragonls edited a comment on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824012171
> This issue is caused by when updated the partition number of topic, triggering `TopicsPartitionChangedListener` look up the new partition number of topic, but `Proxy Client` do not have been granted `LOOKUP`
@linlinnn Is there any solution or suggestion for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] linlinnn commented on issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
linlinnn commented on issue #10300:
URL: https://github.com/apache/pulsar/issues/10300#issuecomment-824913686
Emm.. If the permission granularity is partition, you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #10300: Failed to produce/consumer partitioned topic if updated the topic partition number
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #10300:
URL: https://github.com/apache/pulsar/issues/10300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org