You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mustafasrepo (via GitHub)" <gi...@apache.org> on 2023/02/15 06:47:08 UTC
[GitHub] [arrow-datafusion] mustafasrepo opened a new pull request, #5286: Linear search support for Window Group queries
mustafasrepo opened a new pull request, #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286
# Which issue does this PR close?
Closes [5213](https://github.com/apache/arrow-datafusion/issues/5213).
Closes [5275](https://github.com/apache/arrow-datafusion/issues/5275).
# Rationale for this change
This is a continuation of [#4989](https://github.com/apache/arrow-datafusion/pull/4989), it extends amortized linear search support to window frames in GROUPS mode. As discussed in the preceding PR, this results in significant performance gains, which is the main purpose of this PR.
# What changes are included in this PR?
This PR removes the old GROUPS mechanism and replaces it with the amortized linear search. This reduces code size and yields performance gains simultaneously.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #5286: Linear search support for Window Group queries
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286#discussion_r1108917882
##########
datafusion/proto/src/logical_plan/from_proto.rs:
##########
@@ -907,16 +908,13 @@ pub fn parse_expr(
.window_frame
.as_ref()
.map::<Result<WindowFrame, _>, _>(|window_frame| {
- let window_frame: WindowFrame = window_frame.clone().try_into()?;
- if WindowFrameUnits::Range == window_frame.units
- && order_by.len() != 1
- {
- Err(proto_error("With window frame of type RANGE, the order by expression must be of length 1"))
- } else {
- Ok(window_frame)
- }
+ let window_frame = window_frame.clone().try_into()?;
+ regularize(window_frame, order_by.len())
Review Comment:
I don't understand why `regularize` is called in the serialization code. It makes sense to be called as part of the SQL planner (or LogicalPlanBuilder) so that by the time the structures were serialized / deserialized they should already be valid (and if they are not it should be an error).
##########
datafusion/proto/src/logical_plan/from_proto.rs:
##########
@@ -907,16 +908,13 @@ pub fn parse_expr(
.window_frame
.as_ref()
.map::<Result<WindowFrame, _>, _>(|window_frame| {
- let window_frame: WindowFrame = window_frame.clone().try_into()?;
- if WindowFrameUnits::Range == window_frame.units
- && order_by.len() != 1
- {
- Err(proto_error("With window frame of type RANGE, the order by expression must be of length 1"))
- } else {
- Ok(window_frame)
- }
+ let window_frame = window_frame.clone().try_into()?;
+ regularize(window_frame, order_by.len())
})
- .transpose()?.ok_or_else(||{DataFusionError::Execution("expects somothing".to_string())})?;
+ .transpose()?
+ .ok_or_else(|| {
+ DataFusionError::Execution("expects something".to_string())
Review Comment:
```suggestion
DataFusionError::Execution("missing window frame during deserialization".to_string())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on a diff in pull request #5286: Linear search support for Window Group queries
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on code in PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286#discussion_r1109034121
##########
datafusion/proto/src/logical_plan/from_proto.rs:
##########
@@ -907,16 +908,13 @@ pub fn parse_expr(
.window_frame
.as_ref()
.map::<Result<WindowFrame, _>, _>(|window_frame| {
- let window_frame: WindowFrame = window_frame.clone().try_into()?;
- if WindowFrameUnits::Range == window_frame.units
- && order_by.len() != 1
- {
- Err(proto_error("With window frame of type RANGE, the order by expression must be of length 1"))
- } else {
- Ok(window_frame)
- }
+ let window_frame = window_frame.clone().try_into()?;
+ regularize(window_frame, order_by.len())
Review Comment:
Agreed. That logic was already here from before (in addition to being elsewhere too). In this PR we refactored it out into the `regularize` function as a first step, so at least there is no code duplication anymore. This way, we will be able to remove it easily later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on a diff in pull request #5286: Linear search support for Window Group queries
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on code in PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286#discussion_r1109034121
##########
datafusion/proto/src/logical_plan/from_proto.rs:
##########
@@ -907,16 +908,13 @@ pub fn parse_expr(
.window_frame
.as_ref()
.map::<Result<WindowFrame, _>, _>(|window_frame| {
- let window_frame: WindowFrame = window_frame.clone().try_into()?;
- if WindowFrameUnits::Range == window_frame.units
- && order_by.len() != 1
- {
- Err(proto_error("With window frame of type RANGE, the order by expression must be of length 1"))
- } else {
- Ok(window_frame)
- }
+ let window_frame = window_frame.clone().try_into()?;
+ regularize(window_frame, order_by.len())
Review Comment:
Agreed. That logic was already here from before, in this PR we just refactored it out into the `regularize` function so that we will be able to remove it easily later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #5286: Linear search support for Window Group queries
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286#issuecomment-1434742309
Benchmark runs are scheduled for baseline = ded897ee7f4944ab5ef266854e000179650b5f07 and contender = 79ecf949a489485444825cc19af7ec0bea09452e. 79ecf949a489485444825cc19af7ec0bea09452e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/f362f55f83bf4d968d8e545297791bbb...8212aaa97187490a8c67864dc86cccf0/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/59068d42e3ce47738a7f59c04eade0e6...a1353c5033e04c9d850ff291bc7c4874/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/cf3ce5bf78af470f9094c96a30f54f3c...1e6c9265f4c6402f95a3b85747001fa9/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/1730fe1c137c4e0594f280d07ac61307...81f2729c1fd24f50b002d8cdb5a2114c/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on pull request #5286: Linear search support for Window Group queries
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286#issuecomment-1433747735
Argh, using GH to incorporate the suggestions resulted in a formatting error, just sent a manual commit again. Feel free to merge after CI passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #5286: Linear search support for Window Group queries
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #5286:
URL: https://github.com/apache/arrow-datafusion/pull/5286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org