Re: [I] [multistage] Null Check Predicates Not Pushed Down [pinot]

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on issue #10381:
URL: https://github.com/apache/pinot/issues/10381#issuecomment-1792021791

   News here:
   * https://github.com/apache/pinot/pull/11824 is being reviewed. This PR tries to bring column level nullability by changing the way null vector index is being read. If config indicates the index should not be read, null is returned and semantically V1 query engine behaves as there were no null (although the plan is not modified and therefore more expensive than when `SET enableNullHandling=false`.
   * There is another PR (https://github.com/apache/pinot/pull/11935) that tries to implement the same feature but using another approach. Instead of changing whether the index is loaded or not, it adds a new query option that has to be enabled to use column level nullability. In case it is enabled, V1 will verify whether FieldSpec.isNullable() is false. In that case it won't use the null value vector. Again, plans are not changed. This solution is lighter in terms of semantic changes and the modification is simple (changing a boolean with an enum). But given the null handling code is very repetitive, it requires to apply a simple change in a lot of places, modifying more than 1k lines of code.
   * I've modified both PRs to add a table showing the semantics. I think they make them easier to understand.
   * @walterddr and I discussed about both PRs and considered that https://github.com/apache/pinot/pull/11935 is easier to understand, but we are open to other points of view


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org