You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "deemoliu (via GitHub)" <gi...@apache.org> on 2023/12/13 09:27:11 UTC
[I] Enhance implicit partitioned replica-group assignment behavior for Pinot Upsert [pinot]
deemoliu opened a new issue, #12146:
URL: https://github.com/apache/pinot/issues/12146
Currently for Upsert tables,
`Using implicit partitioned replica-group assignment from low-level consumer won't persist the instance assignment (mapping from partition to servers) to the ZooKeeper, and new added servers will be automatically included without explicit reassigning instances (usually through rebalance). `
To provide an example, we create a Upsert table with BalanceNumSegmentAssignmentStrategy (2 replicas), on a 4 nodes tenant. the partitions can be assigned to
```
Partition0: server0, server1
Partition1: server2, server3
Partition2: server0, server1
Partition3: server2, server3
```
When adding one extra server without rebalancing the table, we started to see
```
Partition0: server0, server1, newServer
Partition1: server2, server3
Partition2: server0, server1, newServer
Partition3: server2, server3
```
The newServer hosting primaryKeys of partition0 but not all the primarykeys are hosted on newServer, and it will failed to lookup the primary keys during ingestion, and duplicates keys and incorrect query results.
The concerns of using implicit partitioned replica-group assignment is, adding new node and rebalancing the table are not atomic operations. After a tenant expansion and before the table get rebalanced, we will see incorrect result for Upsert table.
Is there any reason/scenarios that we need the current behavior of the implicit assignment?
Shall we change the implicit assignment behavior to be the same as the explicit assignment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Enhance implicit partitioned replica-group assignment behavior for Pinot Upsert [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12146:
URL: https://github.com/apache/pinot/issues/12146#issuecomment-1858421799
This issue itself is addressed by #11628. But I'm +1 on only allow explicit replica-group assignment for upsert table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Enhance implicit partitioned replica-group assignment behavior for Pinot Upsert [pinot]
Posted by "deemoliu (via GitHub)" <gi...@apache.org>.
deemoliu commented on issue #12146:
URL: https://github.com/apache/pinot/issues/12146#issuecomment-1853555969
cc: @yupeng9 @Jackie-Jiang @rohityadav1993 @MeihanLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org