You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/10 05:41:40 UTC
[GitHub] [arrow-datafusion] Jimexist opened a new pull request #532: use logical planner in ballista building from protobuf
Jimexist opened a new pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532
# Which issue does this PR close?
Closes #.
# Rationale for this change
use logical planner in ballista building from protobuf, to reduce code duplication
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#issuecomment-858336030
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#532](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4b29444) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/d5bca0e350d94a1e1063bed8a0da0cb09c6e3e1c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5bca0e) will **increase** coverage by `0.09%`.
> The diff coverage is `65.38%`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #532 +/- ##
==========================================
+ Coverage 76.03% 76.12% +0.09%
==========================================
Files 157 157
Lines 26990 26941 -49
==========================================
- Hits 20521 20510 -11
+ Misses 6469 6431 -38
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ta/rust/core/src/serde/physical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9waHlzaWNhbF9wbGFuL2Zyb21fcHJvdG8ucnM=) | `39.83% <47.05%> (+2.11%)` | :arrow_up: |
| [datafusion/src/physical\_plan/planner.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9wbGFubmVyLnJz) | `80.32% <100.00%> (+0.12%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d5bca0e...4b29444](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#issuecomment-858751549
I will have time tomorrow to review this and other pending ballista PRs
On Thu, Jun 10, 2021, 9:47 AM QP Hou ***@***.***> wrote:
> ***@***.**** approved this pull request.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-datafusion/pull/532#pullrequestreview-680957164>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHEBRANXOWJAIYJNOMKQTTTSDNCVANCNFSM46NLUHWQ>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter edited a comment on pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#issuecomment-858336030
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#532](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d0f2804) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/8f84564edab1679163d91691f63381f38907d515?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8f84564) will **increase** coverage by `0.12%`.
> The diff coverage is `66.21%`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #532 +/- ##
==========================================
+ Coverage 76.04% 76.17% +0.12%
==========================================
Files 157 156 -1
Lines 27098 27017 -81
==========================================
- Hits 20608 20580 -28
+ Misses 6490 6437 -53
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ta/rust/core/src/serde/physical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9waHlzaWNhbF9wbGFuL2Zyb21fcHJvdG8ucnM=) | `39.83% <47.05%> (+2.11%)` | :arrow_up: |
| [datafusion/src/physical\_plan/planner.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9wbGFubmVyLnJz) | `78.79% <61.90%> (-1.41%)` | :arrow_down: |
| [datafusion/src/physical\_plan/windows.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi93aW5kb3dzLnJz) | `86.46% <100.00%> (+0.24%)` | :arrow_up: |
| [...sta/rust/core/src/serde/logical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9sb2dpY2FsX3BsYW4vZnJvbV9wcm90by5ycw==) | `35.04% <0.00%> (-0.17%)` | :arrow_down: |
| [datafusion/src/physical\_plan/window\_frames.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi93aW5kb3dfZnJhbWVzLnJz) | | |
| [datafusion/src/sql/planner.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvc3FsL3BsYW5uZXIucnM=) | `84.85% <0.00%> (+0.23%)` | :arrow_up: |
| [datafusion/src/logical\_plan/window\_frames.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvbG9naWNhbF9wbGFuL3dpbmRvd19mcmFtZXMucnM=) | `89.39% <0.00%> (+2.78%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8f84564...d0f2804](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] edrevo commented on a change in pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
edrevo commented on a change in pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#discussion_r649800622
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
I just realized that this PR changes the behavior when there is a non-empty partition_by/order_by/window_frame: before (at least in ballista) it would error, whereas now it is silently ignored. Maybe it is worth erroring is they aren't empty to make it explicit that there is no support?
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
nit: once this is merged, it would be cool to add a link to this code in the respective issues that are tracking these missing features to make it easier for new contributors to start contributing.
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
I just realized that this PR changes the behavior when there is a non-empty partition_by/order_by/window_frame: before (at least in ballista) it would error, whereas now it is silently ignored. Maybe it is worth erroring if they aren't empty to make it explicit that there is no support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on a change in pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
Jimexist commented on a change in pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#discussion_r649843237
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
it'll be further developed in https://github.com/apache/arrow-datafusion/pull/520
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
but that's a good point let me add guard here.
##########
File path: datafusion/src/physical_plan/windows.rs
##########
@@ -61,24 +63,27 @@ pub struct WindowAggExec {
/// Create a physical expression for window function
pub fn create_window_expr(
fun: &WindowFunction,
+ name: String,
args: &[Arc<dyn PhysicalExpr>],
+ _partition_by: &[Arc<dyn PhysicalExpr>],
+ _order_by: &[PhysicalSortExpr],
+ _window_frame: WindowFrame,
Review comment:
this is fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter edited a comment on pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532#issuecomment-858336030
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#532](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6ff2361) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/d5bca0e350d94a1e1063bed8a0da0cb09c6e3e1c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5bca0e) will **increase** coverage by `0.07%`.
> The diff coverage is `64.28%`.
[](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #532 +/- ##
==========================================
+ Coverage 76.03% 76.10% +0.07%
==========================================
Files 157 157
Lines 26990 26976 -14
==========================================
+ Hits 20521 20531 +10
+ Misses 6469 6445 -24
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ta/rust/core/src/serde/physical\_plan/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YmFsbGlzdGEvcnVzdC9jb3JlL3NyYy9zZXJkZS9waHlzaWNhbF9wbGFuL2Zyb21fcHJvdG8ucnM=) | `39.83% <47.05%> (+2.11%)` | :arrow_up: |
| [datafusion/src/physical\_plan/planner.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi9wbGFubmVyLnJz) | `78.79% <61.90%> (-1.41%)` | :arrow_down: |
| [datafusion/src/physical\_plan/windows.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvcGh5c2ljYWxfcGxhbi93aW5kb3dzLnJz) | `86.46% <100.00%> (+0.24%)` | :arrow_up: |
| [datafusion/src/logical\_plan/window\_frames.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/532/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9zcmMvbG9naWNhbF9wbGFuL3dpbmRvd19mcmFtZXMucnM=) | `88.39% <0.00%> (+1.78%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d5bca0e...6ff2361](https://codecov.io/gh/apache/arrow-datafusion/pull/532?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #532: reuse datafusion physical planner in ballista building from protobuf
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #532:
URL: https://github.com/apache/arrow-datafusion/pull/532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org