You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/14 21:39:04 UTC
[GitHub] [incubator-pinot] npawar opened a new issue #5390: Improve the experience when adding indexes and reloading
npawar opened a new issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390
When inverted indexes are added to an existing table, a `POST /segments/{tableName}/reload` needs to be issued to apply it. This call immediately returns 200, and it is not easy to figure out the status of the reload or the indexing.
Some ways we have been using:
1) Untar segment to check if index is available in index_map
2) Check the numEntriesScanned for a query with filter on that column
However, we need a proper way. One way could be to introduce a segment API to read metadata from the segment dir
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar closed issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
npawar closed issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] guruguha commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
guruguha commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-637918220
I have an initial approach to this issue:
We can introduce a segment state monitor that keeps track of all the segments and their index version updates. Similar to `_lastKnownSegmentMetadataVersionMap`, but it is updated whenever a segment index is updated. The monitor could be the callback passed as part of the segment reload - currently there is none.
On the API front,
- provide an end point for users to query for segment status
- get the version change status from the resource manager
- parse the SegmentZKMetadata object to return the value as a JSON string
Please let me know your thoughts and inputs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] guruguha commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
guruguha commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-637914979
I have an initial approach to this issue:
- provide an end point for users to query for segment status
- the end point would need to query the ZK to get the input segment metadata based on the type of the table
- parse the `SegmentZKMetadata` object to return the value as a JSON string
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-638303323
based on our offline chat @guruguha
```the only option is to contact server. The controller can read the external view to figure out servers which have the segment, and server can have an endpoint for returning info about the segment
I think TableSizeReader does something similar. Server api in TableSizeResource for reference
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-637941159
I think one approach could be as simple as:
We introduce an API for reading physical segment's metadata files `/segments/index-metadata` (need a better name)
1. controller untars the segment
2. reads the metadata using SegmentMetadataImpl
3. extracts needed info from metadata.properties and indexMap
4. deletes uncompressed segment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] guruguha commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
guruguha commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-643821124
I have a Google doc with the description and base design. Please review: Pinot [#5390](https://docs.google.com/document/d/1E_J7PxF9WtaE6ido__u0O-Emtyu0IghQicC-1eLZeRo)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar edited a comment on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
npawar edited a comment on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-638303323
based on our offline chat @guruguha
```
the only option is to contact server. The controller can read the external view to figure out servers which have the segment, and server can have an endpoint for returning info about the segment
I think TableSizeReader does something similar. Server api in TableSizeResource for reference
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar edited a comment on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
npawar edited a comment on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-638303323
based on our offline chat @guruguha
the only option is to contact server. The controller can read the external view to figure out servers which have the segment, and server can have an endpoint for returning info about the segment
I think TableSizeReader does something similar. Server api in TableSizeResource for reference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] guruguha removed a comment on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
guruguha removed a comment on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-637914979
I have an initial approach to this issue:
- provide an end point for users to query for segment status
- the end point would need to query the ZK to get the input segment metadata based on the type of the table
- parse the `SegmentZKMetadata` object to return the value as a JSON string
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-638305405
Yes, contact the server is the best. Please put up an API spec (either in this issue or in a separate doc) and send out email (and perhaps on slack as well) for people to comment.
thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5390: Improve the experience when adding indexes and reloading
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5390:
URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-638294063
How about introducing an API on the server, since that is what we are looking for. The controller can query each replica about whether indexes are loaded correctly, which version (crc) is loaded, etc. We can also keep some statistics in the server that can be retuned about the segment (num rows scanned per query, number of queries that the segment hit, whatever..)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org