You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/03 21:34:47 UTC
[GitHub] [incubator-pinot] pabrahamusa opened a new issue #6539: Reducing Partition count - server does not host this topic-partition
pabrahamusa opened a new issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539
I had 4 partitions and 3 Pinot servers running. And noticed that only 3 partitions are getting consumed as I only have 3 servers. So I reduced the partition count from 4 to 3 in kafka. After that I am seeing errors in one of the Pinot server, other servers are fine. Restarts not fixing this issue.
```
my-pinot-server-2 server 2021/02/03 21:22:40.400 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.503 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.605 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.707 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.809 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:40.911 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.014 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.116 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.218 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.328 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.430 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.532 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
my-pinot-server-2 server 2021/02/03 21:22:41.635 ERROR [ConsumerCoordinator] [mylog__3__0__20210203T1927Z] [Consumer clientId=consumer-3, groupId=] Offset commit failed on partition my_all_logs-3 at offset 0: This server does not host this topic-partition.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa closed issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa closed issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-772872275
You mean Pinot doesn't assign 4 consuming segments to all 3 servers? If not, it's a problem, we should fix it. Can you share the idealstate/externalview for that?
Pinot should assign 4 partitions from `mylog__0_...` to `mylog__3_...`
Also, shrinking Kafka partitions is not supported, only increase the partition. Need to delete the table and recreate it.
Maybe you can try to increase Kafka partitions from 4 to 6?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-773353878
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa closed issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa closed issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-772961312
> Thanks @fx19880617 for the suggestions. I tried Ideal state and can see all the 4 partitions are getting consumed as shown below which is good. However there is an issue when you shrink Kafka partition i.e delete and recreate a topic with less number of partition than before. When that happens Pinot server keeps on ERROR-ing. Even though everything works the error is cluttering the console also not sure whether this have any other performance impact.
>
> ```
> "mylog__0__314__20210203T1856Z" : {
> "Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
> },
> "mylog__1__7__20210204T0043Z" : {
> "Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
> },
> "mylog__2__7__20210204T0043Z" : {
> "Server_my-pinot-server-1.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
> },
> "mylog__3__0__20210203T1927Z" : {
> "Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
> }
> ```
By design, Pinot is not handling the shrinking case. I think you need to manually delete the consuming segment `mylog__3__0__20210203T1927Z`. Also delete topic and recreate may mess up the consumer offset, @npawar may have more saying about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-773353878
@mcvsubbu , Just want to check . So that means we can only scale up. And no ways to scale down atm as per design.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-772968419
If you change the topic (or recreate it with diffreent partitions) then you need to drop and recreate the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-772942895
Thanks @fx19880617 for the suggestions. I tried Ideal state and can see all the 4 partitions are getting consumed as shown below which is good. However there is an issue when you shrink Kafka partition i.e delete and recreate a topic with less number of partition than before. When that happens Pinot server keeps on ERROR-ing. Even though everything works the error is cluttering the console also not sure whether this have any other performance impact.
```
"mylog__0__314__20210203T1856Z" : {
"Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
},
"mylog__1__7__20210204T0043Z" : {
"Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
},
"mylog__2__7__20210204T0043Z" : {
"Server_my-pinot-server-1.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
},
"mylog__3__0__20210203T1927Z" : {
"Server_my-pinot-server-2.my-pinot-server-headless.logging.svc.cluster.local_8098" : "CONSUMING"
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] pabrahamusa commented on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
pabrahamusa commented on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-773721572
Closing ticket as this is not supported feature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #6539: Reducing Partition count - server does not host this topic-partition
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #6539:
URL: https://github.com/apache/incubator-pinot/issues/6539#issuecomment-772872275
You mean Pinot doesn't assign 4 consuming segments to all 3 servers? If not, it's a problem, we should fix it. Can you share the idealstate/externalview for that? We should expect Pinot to assign 4 partitions from `mylog__0_...` to `mylog__3_...`
Also, shrinking Kafka partitions is not supported, only increase the partition. Need to delete the table and recreate it.
Maybe you can try to increase Kafka partitions from 4 to 6?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org