You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/11/22 01:53:10 UTC
[I] Fix large partition id for single stream [pinot]
Jackie-Jiang opened a new issue, #17260:
URL: https://github.com/apache/pinot/issues/17260
When multiple stream support was added in #13790, it assumes the partition id for a given stream can never go over `10000`. This assumption cannot be held for certain custom stream plugins.
#15957 fixed the problem by always checking if it is single stream before applying the modulo operation in `IngestionConfigUtils` (`getStreamPartitionIdFromPinotPartitionId` and `getStreamConfigIndexFromPinotPartitionId`).
Several recent PRs break this again, and we need to fix it:
- #16783 @noob-se7en
- #16833 @lnbest0707-uber
- #17163 @noob-se7en
- #17217 @tarun11Mavani
In the meanwhile, we should consider adding a test for this scenario.
@jadami10 Please confirm if you are still using a plugin that can generate large partition id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "lnbest0707-uber (via GitHub)" <gi...@apache.org>.
lnbest0707-uber commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3630265360
> > But this does not fix the actual problem right?
>
> Agree. This is just a workaround to not break single stream case, which is much more common than multiple streams.
>
> > I am not sure if the padding approach is the best. Considering that the issue is with the LLCSegmentName naming convention here, maybe we should explore a different naming strategy for multi-stream tables.
>
> +1 on this. The root problem is to overloading partition id to represent multiple streams. cc [@lnbest0707-uber](https://github.com/lnbest0707-uber) [@ankitsultana](https://github.com/ankitsultana) Given the above problems, what's your opinion?
There was one [proposal ](https://github.com/apache/pinot/pull/16494)to add topic name into the segment name but the community were a bit hesitated with it. On one side, it introduced many changes to the existing code. On the other side, it may increase the length of segment name and size of ideal state. Especially the ideal state, it could be very large with the uncontrolled kafka topic names.
Maybe the solution could be slightly changing the segment naming format + keeping the index of the topic only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3578534669
@tarun11Mavani : can you review this and confirm if our internal use-case can continue to work on master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3608649432
> When multiple stream support was added in https://github.com/apache/pinot/pull/13790, it assumes the partition id for a given stream can never go over 10000. This assumption cannot be held for certain custom stream plugins.
I was unaware that the above assumption was wrong while creating the follow up PRs and went ahead based on this comment:
https://github.com/apache/pinot/blame/8bfb907db9eeb29ea3f494f435aa2fa781378db1/pinot-spi/src/main/java/org/apache/pinot/spi/utils/IngestionConfigUtils.java#L58-L60
```
// For partition from different topics, we pad then with an offset to avoid collision. The offset is far higher
// than the normal max number of partitions on stream (e.g. 512).
public static final int PARTITION_PADDING_OFFSET = 10000;
```
> https://github.com/apache/pinot/pull/15957 fixed the problem by always checking if it is single stream before applying the modulo operation in
But this does not fix the actual problem right?
I am not sure if the padding approach is the best. Considering that the issue is with the LLCSegmentName naming convention here, maybe we should explore a different naming strategy for multi-stream tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3567038805
Yes, our plugin uses the full 32 bit range, so we have numbers much greater than 10,000.
Cc @priyen-stripe since you've been looking at an upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3593944552
Can we also add a test scenario for the topic data migration? This also happens many time in the production, e.g. topic migration from large number of partitions to smaller number.
Assume we are switching kafka topic from topicA to topicB:
0. Create both kafka topics: topicA(128 partitions) and topicB(64 partitions), the table is setup with topicA.
For seamless, no downtime nor data loss migration:
1. Add Kafka topicB into the table stream conf, ensure it's existence but no data coming so far
2. Cut the kafka producer from topicA to topicB
3. Wait for topicA data persisted in the table
4. Remove topicA from the stream configs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-6377451520
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3565334128
See #17261 for example of fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3629599550
> But this does not fix the actual problem right?
Agree. This is just a workaround to not break single stream case, which is much more common than multiple streams.
> I am not sure if the padding approach is the best. Considering that the issue is with the LLCSegmentName naming convention here, maybe we should explore a different naming strategy for multi-stream tables.
+1 on this. The root problem is to overloading partition id to represent multiple streams. cc @lnbest0707-uber @ankitsultana Given the above problems, what's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-3633713558
yeah, was thinking something similar like storing all the topic names of the table in ZkMetadata as list and using the index of those topic names while generating the segment name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-4503798694
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Fix large partition id for single stream [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #17260:
URL: https://github.com/apache/pinot/issues/17260#issuecomment-4515283555
bumping this up as its still relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org