[GitHub] [pinot] kishoreg commented on pull request #7820: push JSON Path evaluation down to storage layer

[GitBox <gi...@apache.org>]

kishoreg commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-982088481


   I like the concept of the pushdown evaluator. Few thoughts/concerns
   - Some methods added to projectionBlock seem to be specific to projectionBlock and is deviating from the block concept of chaining any blocks together i.e. the caller of the block does not understand the implementation of the underlying block. We have violated this in few places and we should fix that. It might be a good idea to enhance the Block interface to allow a pushdown evaluator.
   - Conceptually, we are moving from a Column to Column + path but modelling the path as an evaluator. But the query layer is deciding the evaluator instead of the storage layer. 
   - The evaluator seems to be set at the JVM level but in practice, it's possible for each table to have a different evaluator and in fact, it's possible that each segment can have a different evaluator.
   
   At a high level, the segment is the smallest unit of query processing in Pinot and most of the things can be decided independently at that scope - segmentDirectory, readers, writers, Blocks, indexes, encoding etc.
   
   Having said that, I don't have a very good solution yet but I think we can come up with something after a couple of iterations. One thought is enhancing the concept (String Column) to a Column Object which has  SingleColumn, NestedColumn( column + path) as implementations.
   
   Overall, I think the Storage (Reader) should decide on the evaluator instead of the query layer. 
   
    
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org