You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/10/19 03:16:16 UTC
[GitHub] [pinot] npawar opened a new issue #7594: Remove dependence of strict naming convention for realtime segments
npawar opened a new issue #7594:
URL: https://github.com/apache/pinot/issues/7594
The realtime segments are named as "tableName__partitionId__sequenceNum__timestamp".
This convention makes it harder to perform operations such as table migration, adhoc uploads, etc.
In other intiatives such as pluggable streams, we have encountered issues with the presence of timestamp in segment name.
It would be nice to get to a state where we dont depend on the segment name having a convention as such. For starters, we can make this change for completed segments, and then trickle it down to consuming as well.
cc @mayankshriv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on issue #7594: Remove dependence of strict naming convention for realtime segments
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #7594:
URL: https://github.com/apache/pinot/issues/7594#issuecomment-946887621
It may take a few releases, but yes, I am supportive of this.
I think the root is perhaps in the segment completion protocol, where table name can be sent as an additional argument as opposed to deriving it from the segment name. Once we introduce a new protocol element, then we can cut an incompat release where the controller expects the server to send this new protocol element. I suggest checking this in soon, and have the controller changes to get the table name from the protocol element if it is available, or else from the segment name. Over one or two releases, we can eliminate the latter.
Another part of the naming convention is the double underscore part. This is a harder thing to remove, IMO. We need the sequence number and partition ID for sure. The timestamp, while being optional, has come in super handy for debugging in production environments, and also while simply looking at the zookeeper via a client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #7594: Remove dependence of strict naming convention for realtime segments
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #7594:
URL: https://github.com/apache/pinot/issues/7594#issuecomment-946333734
+1, especially the `getTableName()` api that gets table name from `segmentName` makes it very rigid. In the offline side, we have decoupled table name from the segment metadata as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org