Re: [I] [EPIC] Implement Range Partitioning [datafusion]

["stuhood (via GitHub)" <gi...@apache.org>]

stuhood commented on issue #22395:
URL: https://github.com/apache/datafusion/issues/22395#issuecomment-4618822694

   @gene-bordegaray : 
   1. today's discussion (thanks!)
   2. the design doc that I've been working on
   3. reviewing #22657
   
   ...have highlighted one of the tradeoffs of the current definition of Range partitioning that I wanted to call out:
   
   ----
   
   As defined, Partitioning::Range contains no overlapping partitions. This is aligned with Partitioning::Hash (I can't really imagine what overlap would mean there), but allowing for overlapping _output_ partitioning from a `TableProvider`, while continuing to require that _input_ partitioning is total and non-overlapping (is this a `enum Distribution`...?) would have some really interesting properties.
   
   In order to produce a reasonable number of Range partitions _which match the other side of the join_, the underlying data will either:
   1. `DISJOINT` - already need to be over-partitioned into a series of disjoint ranges, in which case the creator of the `TableProvider` would merge/aggregate a few of the underlying files/partitions into the declared Ranges (e.g. go from 1000 underlying partitions to 16 declared Partitioning::Range partitions)
   2. `NOT-DISJOINT` - if the underlying files/partitions are _not_-disjoint / are-overlapping, then the `TableProvider` will need to:
       1. dynamically choose partition boundaries (using statistics about the amount of overlap, size, etc)
       2. bucket the files into those partitions 
       3. push down or execute range filters for cases where a file spans a partition boundary (to guarantee that data is emitted in exactly one partition)
   
   ----
   
   All of this is fine, and we're prepared to do it in our implementation. But if `Partitioning::Range` supported overlapping output partitions, things would look a little bit different:
   
   In that case instead, a wrapper (?) around the `TableProvider` might implement the steps from the `NOT-DISJOINT` case above: it would take the underlying overlapped partitions, consume their statistics (generically), and push down static per-file filters based on the chosen boundaries, and then coalesce their outputs.
   
   The benefit of this would be one upstream implementation of this range filtering... and possibly also some amount of sharing of the coordination required _between_ the two sides of a join, which we have been planning to implement with an optimizer rule inserted at the appropriate spot to inspect both sides of the join, and align their Partitioning::Range declaration based on their shared statistics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org