You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/14 22:03:05 UTC
[GitHub] [incubator-pinot] npawar opened a new issue #5391: Update segment refresh time in segment metadata on reload
npawar opened a new issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391
When segments are reloaded, the segment refresh time is not updated in the segment metadata. There's no other property that would indicate reload time either.
Either start updating refresh time on segment reload, or add a new property to indeicate the last reload time of segments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5391: Provide an indication that segments reload is complete
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629713565
It’s not about changing metadata in ZK, it’s about the response of metadata call on the server. We don’t have a way to know anything about actual contents of segment
- what r the columns
- their encoding
- indexes
Anything else that we need to know about the segment on the server
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5391: Provide an indication that segments reload is complete
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629713889
This could just be included in this: https://github.com/apache/incubator-pinot/issues/5390 to begin with.
@reallocf is taking 5390 up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5391: Update segment refresh time in segment metadata on reload
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629520902
> IMO, servers should not change the segment ZK metadata. It should only be managed by the controller. We have multiple replicas of the same segment on different servers, and segment can be moved to any server at any time. If we allow servers to change the ZK metadata, that will be very hard to manage.
> I think providing API for querying segment metadata is okay. We can return the segment metadata for each replica of a segment.
+1
The data of whether servers loaded inverted index (or any other) should be with the servers.
One way of doing this is to raise a metric on the servers when reload fails, and set alert on the metric. In theory, the reload can take infinite time to arrive and therefore it is never possible to say when it is "done". In practice, however, it should be a short period of time for the message itself to be transmitted to the servers, so we should be ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5391: Update segment refresh time in segment metadata on reload
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629713263
I think the title and description are focused more on the how instead of the why. This is mainly trying to solve the problem - user did a reload (for whatever reason, usually inv index), how to indicate that reload is done. How we do it is implementation details.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5391: Update segment refresh time in segment metadata on reload
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629517968
IMO, servers should not change the segment ZK metadata. It should only be managed by the controller. We have multiple replicas of the same segment on different servers, and segment can be moved to any server at any time. If we allow servers to change the ZK metadata, that will be very hard to manage.
I think providing API for querying segment metadata is okay. We can return the segment metadata for each replica of a segment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5391: Update segment refresh time in segment metadata on reload
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629515069
Some context - when users add inverted indexes and reload, we provide no indication whether reload was successful or not. This is one of the things that can help. Other option is we provide an API for displaying segment metadata (from segment dir)- https://github.com/apache/incubator-pinot/issues/5390
I think both are good to have
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5391: Update segment refresh time in segment metadata on reload
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #5391:
URL: https://github.com/apache/incubator-pinot/issues/5391#issuecomment-628952160
I would doubt about this extra timestamp. Segment refresh time is used to track when the segment is refreshed (replaced by a new segment), while reloading is just some server behavior (can be triggered by controller though). When servers restart, they will also reload all the segments. I don't think it is necessary to track all these timestamps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org