You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/08/07 04:06:18 UTC
[GitHub] [incubator-pinot] jackjlli opened a new pull request #5826: Identify if the values are converted from map and emit server metric
jackjlli opened a new pull request #5826:
URL: https://github.com/apache/incubator-pinot/pull/5826
## Description
Currently there are some existing use cases who use map structure for columns in the raw data during segment creation, while map structure support isn't completely done yet. Thus, these use cases happen to use the undefined behavior in Pinot and thus need to be identified.
And there is no monitoring/alerting system during segment creation.
This PR introduces a way to detect whether the input data are converted from a map to an array.
The info is saved in the segment metadata, so that pinot-server can emits a metric to detect this behavior once the segment is downloaded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on pull request #5826: Identify if the values are converted from map and emit server metric
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on pull request #5826:
URL: https://github.com/apache/incubator-pinot/pull/5826#issuecomment-674558001
Seems we are catching just one specific error (input is map, but schema is MV). In general, there can be mis-match due to any number of reasons. We should just catch that and flag. To begin with, we should only flag (perhaps emit a metric), and not enforce (and fail segment generation) as there are likely flows out there where the code is tolerant enough to pass today. Once we have a mechanism for identify such flows, we can then clean them up and then enforce schema validation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli closed pull request #5826: Identify if the values are converted from map and emit server metric
Posted by GitBox <gi...@apache.org>.
jackjlli closed pull request #5826:
URL: https://github.com/apache/incubator-pinot/pull/5826
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org