You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gene-bordegaray (via GitHub)" <gi...@apache.org> on 2026/05/02 19:55:52 UTC
[I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
gene-bordegaray opened a new issue, #21992:
URL: https://github.com/apache/datafusion/issues/21992
I’d like to restart the partitioning side of this discussion separately from the dynamic-filter PRs and threads (see #21207 for more background)
The main issue seems to be that DataFusion cannot always represent the physical partitioning that some data sources actually have. In our case, range-partitioned data has had to claim `Partitioning::Hash`, which lets us avoid repartitioning but makes later optimizer and dynamic-filter decisions brittle due to this false advertisement.
Today we are in this scenario:
```text
Data is actually range-partitioned:
- partition 0: key < 100
- partition 1: 100 <= key < 200
- partition 2: 200 <= key < 300
To mimic this we tell DataFusion our partitioning properties are:
- Partitioning::Hash([key], 3)
```
Those are not the same partitioning scheme.
The general goal should be that DataFusion should be able to describe physical partitioning truthfully, be able to compare two partitionings for compatibility, and use that information only when compatibility is proven.
This is well shown in dynamic filtering:
```text
Build side partitioning:
p0: key < 100
p1: 100 <= key < 200
p2: 200 <= key < 300
Probe side partitioning:
p0: key < 100
p1: 100 <= key < 200
p2: 200 <= key < 300
```
These are compatible as every partition X on the build side is compatible with partition X on the probe side. With this information, optimizations can safely use partition-local behavior:
```text
if partitioning is compatible:
use partition-local filter (more selective)
else:
use global/safe fallback
```
A possible direction is to evolve Partitioning toward an extensible abstraction, such as a `PhysicalPartitioning` trait, and add range partitioning as the first use case. Longer term, built-ins like hash partitioning could move behind the same abstraction.
This would give us a cleaner foundation for:
- range/file partitioned sources
- partition-local dynamic filters when both sides share the same partition map #21832
Does this direction seem interesting to others in the community nd any thoughts on this proposal?
cc: @NGA-TRAN @alamb @jayshrivastava @gabotechs @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4373596563
I think it would also help to decide sooner rather than later if we want a general partitioning trait, or if we are just going to hard code Range Partitioning. I think either could work.
If we are going to go with a trait, it might be good to declare that we eventually want to shoot to remove all special cases for hash partitioning 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4479148755
PR updated according to above. I decided to keep the clearer representation of `Range` partitioning as @alamb described in his message, then we can use the split points under the hood when needed to repartition. I added comments documenting this as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4461685190
Here is the mechanical only PR: #22207 as discussed (both here and in discussions). I have tried to make the contract of the partitioning as clear as possible in the rust docs, please provide feedback on that part as it is critical users understand what they are buying into.
cc: @NGA-TRAN @adriangb @gabotechs @jayshrivastava @stuhood @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4373560671
I left some comments on https://github.com/apache/datafusion/pull/22002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4366614993
I have added a draft PR #22002 which shows my vision for what partitioning can look like going forward and how this will apply to an optimization like dynamic filter routing. The API design was done by me but implementation details by AI and is not meant to be a mergeable PR. I provided a description of the API, would love any suggestions/ideas on this general approach.
I separated it into 4 commits to show the logical progression of the implementation:
1. Adding Range partitioning variant that models what a custom partitioning trait would look like
2. Introduce the general partitioning trait using this model and have range partitioning implement it
3. Have file preserved partitioning use range partitioning rather than false advertising hash
4. Route dynamic filters using partitioning compatibility and create partitioning-local filters when range partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-5411351268
> Given [#22395](https://github.com/apache/datafusion/issues/22395) maybe we should close this issue and track the work there
sounds good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4374132378
> I think it would also help to decide sooner rather than later if we want a general partitioning trait, or if we are just going to hard code Range Partitioning. I think either could work.
I vote for a general partitioning trait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4374956110
The enum Gene proposed looks good. As long as there are PhysicalExpr, it will work
```rust
Range {
exprs: Vec<Arc<dyn PhysicalExpr>>,
partitions: Vec<RangePartition>,
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4434586668
I think that partitioning is likely to become relevant for us in the next few weeks, and our schedule should finally be clearing up to try and get some of our team involved with helping.
We're very likely to be doing either Range or N-dimensional partitioning. AFAICT though, Range is sufficient to represent N-dimensional partitioning (with a physical optimizer rule running before EnforceDistribution), because you can declare each table to be partitioned on the relevant join key...?
So, in practice, it doesn't seem like it would be worth actually introducing knowledge of multi-dimensional ranges here: a Range type would be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4450827153
We spoke about this topic the other day in person when we were at the NYC meetup
I believe @gene-bordegaray 's usecase will be satisfied if we can represent range partitioning that is represented like
* `[expr]`: `[[ranges]]`
For example, partitioning on `date` would be represented like
* `[date]`: [
[[2021-01-01, 2021-12-32]]
[[2022-01-01, 2022-12-32]],
...
]
partitioning on `date, city` would be represented like
* `[date, city]`: [
[[2021-01-01, 2021-12-32], [Allston, Boston]]
[[2021-01-01, 2021-12-32], [Boston, NYC]]
[2022-01-01, 2022-12-32], [Allston, Boston]]
[2022-01-01, 2022-12-32], [Boston, NYC]]
...
]
Some TBD details are:
1. How to ensure the entire range is covered (e.g. what range in the above example handles 1970-01-01)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4455852315
To keep everyone else in this thread up-to-date, we had some discussions this week regarding this topic and have come up with a concrete plan. We are going to start with adding a enum variant for `ExprPartitioning` (name up for debate) rather than introducing a general trait immediately. @alamb described the concept and representation of this well, so I won't reexplain that here.
For implementation, the first PR will be purely mechanical. This will add a `Expr` (or similar) variant to the existing physical `Partitioning` enum, along with the supporting types, but just throw "not implemented" at callsites. This will introduce the concept and give a good idea of the API methods we will need to represent partitioning well.
This will then be followed up with implementing the partitioning features based on the callsites in follow-up PRs, with one of these beeing the dynamic filter routing.
The goal is still to model this in a way that can evolve into a general partitioning abstraction if needed.
I will make the mechanical PR soon and ping here 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4378732353
I am in favor of a general trait as the long-term goal of this work. I think allowing users to implement their own type of partitioning will make DF more powerful in production use cases as I am sure that there will be instances of partitioning that are not captured in `Hash` or `Range` partitioning (just as `Hash` did not fully work for us). Off the top of my head something like a `Value` partitioning would also be useful:
```text
p0: col in ('a', 'd')
p1: col in ('b')
p2: col in ('c')
```
Because of this I think providing another extendible point for people (the trait) will be very high value even if worth the extra effort.
With this said I do think we can create mergeable commits by extending the enum now by supporting `Range` partitioning as @adriangb has described but model it after what our trait will look like. We can treat the trait as the final goal but let `Range` help us define the requirements for that as we pseudo-implement what that trait will look like.
> If we are going to go with a trait, it might be good to declare that we eventually want to shoot to remove all special cases for
> hash partitioning 🤔
And along with this, yes I agree here that we should shoot to encapsulate all partitioning logic behind this trait and no special cases. The optimizer rules and other things should ask if two partitioning are compatible or satisfy one another, not just "is this Hash partitioned" 👍
I see this path as actually being more intuitive once done well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4452796089
In my experience, range partitioning is usually represented with only the partitioning points, which addresses the infinite end cap issue, and guarantees that everything is covered. Each point is exclusive for the range to the left side of the point and inclusive for the range to the right side of the point. So your example would be more like:
To create three ranges (infinitely small to `2021-01-01`, between `2021-01-01` and `2022-12-32`, and `2022-12-32` to infinitely large)
```
`[date]`: [
[2021-01-01]
[2022-12-32],
...
]
```
And the date + city example would be:
```
`[date, city]`: [
[2021-01-01, Allston],
[2021-01-01, Boston],
[2022-12-32, Allston],
[2022-12-32, NYC],
...
]
```
(or something).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4374152246
I'm not sure if it should be a trait or an eum. I generally prefer an explicit enum *if* the trait doesn't generalize well. E.g. if we can't point to an obvious 3rd use case for the trait that an external system could conceivably implement and where everything would work for them, I fear a trait is the wrong choice.
@NGA-TRAN could you share why you prefer a trait to an enum here? Maybe you know some additional partitioning strategies that are commonly used in the database world that I don't know of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4374703267
I don't know that the enum is any different than the trait in terms of representation. I.e. if we can represent it as a trait we can represent it as an enum. The question is more if we want to hard code all of the handled cases or leave it open for users to add more down the road.
I'd defer to @gene-bordegaray as to what an enum version would look like but I think something like this:
```rust
pub enum Partitioning {
RoundRobinBatch(usize),
Hash {
exprs: Vec<Arc<dyn PhysicalExpr>>,
partition_count: usize,
},
Range {
exprs: Vec<Arc<dyn PhysicalExpr>>,
partitions: Vec<RangePartition>,
},
}
pub struct RangePartition {
/// One interval per `Range.exprs` entry.
///
/// For exprs = [time, int_val], this can represent:
/// time in [2026-05-01, 2026-05-07]
/// AND int_val in [1, 100]
pub bounds: Vec<RangeInterval>,
}
pub struct RangeInterval {
pub lower: Option<RangeBound>,
pub upper: Option<RangeBound>,
}
pub struct RangeBound {
pub value: ScalarValue,
pub inclusive: bool,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4564279074
Sounds good -- for the order I would encourage you to start with a .slt test framework and use that to validate end to end behavior
For example you could `range_partitioning.slt` create a table with range partitioning using the method like this:
https://github.com/apache/datafusion/blob/d4fb7efbbddf3fc02067d6a138bec1d9370101f7/datafusion/sqllogictest/src/test_context.rs#L131-L188
Then start testing with simple things like aggregating on the colums where it it ranged, and not ranged, then jouins, etc
I suspect it would make your PRs easier to review / understand what changed if you made an initial one with some basic tests / plans (like `EXPLAIN SELECT ... GROUP BY range_partition_column` and `EXPLAIN SELECT .. GROUP BY non_range_partition`, etc)
Then as you go through and implement the feature, your PRs will also demonstrate what is changing / getting better at the end to end sql level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4374336137
@adriangb : Can you provide an example of enum? If it covers our use cases, we are happy to go with it.
Currently, our range come from 2 columns (or 2 expressions). For example (time, int_val). So we can have these ranges:
- ( [2026.05.01 - 2026.05.07] , [1 - 100] )
- ( [2026.05.01 - 2026.05.07] , [101 - 200] )
- ...
- ( [2026.05.08 - 2026.05.15] , [1 - 500] )
- ( [2026.05.08 - 2026.05.15] , [501 - 1000] )
- ...
As long as they do not overlaps, they are considered as ranges. We can definitely map those into simple ranges or indexes and use those simple ones in DF instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4466649802
Continuing discussion from the PR here with @stuhood and @Dandandan to keep things traceable for everyone involved. Both have raised question on why we decided to introduce `ExprPartitioning` rather than regular `RangePartitioning`. I have pointed to `ExprPartitioning` being more flexible but rememeber we decided to pursue `Range` before `Expr` due to implementation complexity and use cases (sorry have had lots going on this week 😅 ).
With this I have revisited the white boarding we did and thought about the points brought up on the PR regarding repartitioning.
One option I am thinking about is a range representation that stores the partitioning expressions, the split points for each expression, and a map from range-grid cells to output partitions:
```text
p0: date in [2021-01-01, 2022-01-01) AND city in [Allston, Boston)
p1: date in [2021-01-01, 2022-01-01) AND city in [Boston, NYC)
p2: date in [2022-01-01, 2023-01-01) AND city in [Allston, Boston)
p3: date in [2022-01-01, 2023-01-01) AND city in [Boston, NYC)
Represented as:
exprs: [date, city]
date split points: [2021-01-01, 2022-01-01, 2023-01-01]
city split points: [Allston, Boston, NYC]
cell -> partition:
([2021-01-01, 2022-01-01), [Allston, Boston)) -> p0
([2021-01-01, 2022-01-01), [Boston, NYC)) -> p1
([2022-01-01, 2023-01-01), [Allston, Boston)) -> p2
([2022-01-01, 2023-01-01), [Boston, NYC)) -> p3
```
This is close to @alamb `[exprs] -> [[ranges]]` suggestion, but change a bit for routing as @stuhood brought to my attention. Andrew’s form represents each partition as a set of ranges while this form stores split points plus a `cell -> partition` map. They are both able to represent the same things, but the this makes repartitioning a bit more efficient I believe (please inform me if I am missing something) since we evaluate the expressions, binary-search each dimension’s split points, then look up the target partition instead of scanning every partition’s range.
We could also keep Andrew’s `[exprs] -> [[ranges]]` representation as the public representation, but build this form described above when routing is needed. This might be nice as I think Andrew's representation is a bit more readable but hides some implementation details from the user.
Sorry for the confusion guys, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-5411335448
Given https://github.com/apache/datafusion/issues/22395 maybe we should close this issue and track the work there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4364833770
Great description. The exact use case at DataDog. We plan to use this a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4563595609
Update on work going forward. I plan to implement follow-up work roughly in this order to continue to lay good foundations and incrementally build up support:
1. Range compatibility and rewrite helpers
- Issue: #22395
2. Preserve Range partitioning through physical optimizer rules
- Issue: #22395
3. Expose source Range partitioning
- Issue: child issue under #22395
4. Use compatible Range partitioning in join planning
- Issue: child issue under #22395
5. Dynamic filter partition-local routing
- Issues: #21207, #21992, child issue under #22395
6. Range repartition execution
- Issue: #22397
7. Planner/API rule that requests Range repartitioning
- Issue: child issue under #22397
- Add the reason to insert range repartitions.
8. Expose Range partitioning across FFI
- Issue: #22394
Please let me know if this ordering doesn't seem like the best direction. Would also love nay help in tackling these 😄
cc: @alamb @NGA-TRAN @stuhood
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4501109459
I have created some sub-issues as work we will do after this.
There is one that is an EPIC which will serve as a parent issue for all the optimizer propagation and what not. I will make sub-issues for each but thought would be good to have that in one traceable place. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #21992: [DISCUSSION] Extending Partitioning to Support More Variants
URL: https://github.com/apache/datafusion/issues/21992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4478694027
Thanks for the details @gene-bordegaray @stuhood and @Dandandan. Sounds good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4450830669
(would the above work for you and your usecse @stuhood ?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
gene-bordegaray commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-4564662866
> Sounds good -- for the order I would encourage you to start with a .slt test framework and use that to validate end to end behavior
Ok noted. I made one small PR which will not require slt test as its just a helper for partitioning we will need. Before making any planning / optimizer changes I will make sure a good slt framework is in place for us to use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Extending Partitioning to Support More Variants [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21992:
URL: https://github.com/apache/datafusion/issues/21992#issuecomment-5416577623
Tracking in https://github.com/apache/datafusion/issues/22395
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org