[GitHub] [incubator-pinot] kishoreg commented on issue #5268: Filter during ingestion

[GitBox <gi...@apache.org>]

kishoreg commented on issue #5268:
URL: https://github.com/apache/incubator-pinot/issues/5268#issuecomment-618723467


   Great discussion so far and valid points. My take is filtering should not even be part of schema definition. This is more of an ingestion config which is part of the table config. should we move it to table config?
   
   The challenges mentioned by @npawar such as pre-filter vs post filter etc still remain. I agree with supporting filtering across multiple columns.
   
   Today source field name extractor is looking at the schema but its possible that we look at the filter spec as well to compute the list of fields to extract from source.
   
   Another option is to add a boolean to fieldSpec isTransient=true  which means we just extract it from source but not store in Pinot.
   
   
   
     


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org