[GitHub] [pinot] Jackie-Jiang commented on pull request #7820: push JSON Path evaluation down to storage layer

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on pull request #7820:
URL: https://github.com/apache/pinot/pull/7820#issuecomment-985152221


   > Could this be handled differently without the cache by reordering the execution graph to deduplicate common subexpressions?
   
   I agree. Ideally we should deduplicate the expression during the query planning, and have only one transform function for each unique expression. Then we can remove the cache and avoid all the unnecessary overhead.
   
   Towards this direction, I agree we should maintain the buffer within the transform layer and push the buffer down through the data fetcher. We should think of a smooth way to change the interface and move to the end state where all the transform functions (with or without an evaluator) can pass down the buffer to fill the values.
   
   For now, we might just add some read methods into the `ProjectionBlock` so that it can be used by the `TransformFunction`? Adding these methods to the `Block` interface might not match our end goal.
   
   Do you see a path how we can move to the end state without making temporary changes to the interfaces?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org