You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/08/28 23:15:03 UTC
[GitHub] [incubator-pinot] icefury71 opened a new issue #5942: Better Table config validation
icefury71 opened a new issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942
As mentioned in #5915 and #5899 we need a better way to validate table config. This includes following checks (not exhaustive):
* Validate column names : should match the ones in schema
* Time column check: name and type should be consistent with schema
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-701001585
> yes, that works as long as the same validations are performed. thanks
Yes, its calling the same validation function underneath.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-703891914
+1
Another use case is the upcoming upsert. There are several expectations for upsert table like primary key in schema, replica group setup, routing by replicas group, partition config, ingestion via LLC etc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696164071
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-685997839
https://github.com/apache/incubator-pinot/issues/5900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700900788
> +1 to a utility to validate, we can add a validate endpoint in the controller
@kishoreg @mcvsubbu Seems like we already have a `/tables/validate` end-point in the Table API. This should be sufficient ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696214712
+1 to a utility to validate, we can add a validate endpoint in the controller
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] buchireddy commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
buchireddy commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-699519126
Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
kishoreg commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696214712
+1 to a utility to validate, we can add a validate endpoint in the controller
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700733890
What I meant is - technically it should be supported (it's not today and we
should probably fix that).
On Tue, Sep 29, 2020 at 12:31 AM Buchi Reddy Busi Reddy <
notifications@github.com> wrote:
> Another use case: Validate that the bloomFilterColumns aren't BYTES type
> since that's not supported at this point.
> cc @kishoreg <https://github.com/kishoreg> @icefury71
> <https://github.com/icefury71>
>
> Looks like this should be supported, skipping validation for this
> particular thing
>
> @icefury71 <https://github.com/icefury71> Do you mean bloomFilterColumns
> supports BYTES columns? From which version/
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700507663>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAFRXV5JMUFTZ7U66D3KOFTSIGEN7ANCNFSM4QOTJTBA>
> .
>
--
Thanks and regards
Chinmay Soman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] buchireddy commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
buchireddy commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700507663
> > Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.
> > cc @kishoreg @icefury71
>
> Looks like this should be supported, skipping validation for this particular thing
@icefury71 Do you mean bloomFilterColumns supports BYTES columns? From which version/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-701024984
We can also add checks on partition config where the table can be partitioned on at most one column
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-701001231
yes, that works as long as the same validations are performed. thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700296507
> Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.
>
> cc @kishoreg @icefury71
Looks like this should be supported, skipping validation for this particular thing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696207582
With all the changes proposed/added, it will be useful if a utility is also provided to validate an existing tableconfig. Many of them are tightening the (lax) rules from before, and are valid. It will be unfortunate if an existing installation suddenly stopped working because code elsewhere (not just in the table addition path) starts assuming things about the tableconfig.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696260153
Updated ticket description with all the suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 edited a comment on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 edited a comment on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-700900788
> +1 to a utility to validate, we can add a validate endpoint in the controller
@kishoreg @mcvsubbu Seems like we already have a `/tables/validate` end-point in the Table API. This should be sufficient ? (Same thing with Schemas : `/schemas/validate`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] buchireddy edited a comment on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
buchireddy edited a comment on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-699519126
Another use case: Validate that the bloomFilterColumns aren't BYTES type since that's not supported at this point.
cc @kishoreg @icefury71
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696207582
With all the changes proposed/added, it will be useful if a utility is also provided to validate an existing tableconfig. Many of them are tightening the (lax) rules from before, and are valid. It will be unfortunate if an existing installation suddenly stopped working because code elsewhere (not just in the table addition path) starts assuming things about the tableconfig.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-683191930
One more: https://github.com/apache/incubator-pinot/issues/5730
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation
Posted by GitBox <gi...@apache.org>.
icefury71 commented on issue #5942:
URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696164071
One more check: Currently retention does not work if pushType is null (for real-time table). Check for a case where pushType is null and retention is configured for a table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org