You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/20 14:31:01 UTC
[GitHub] [arrow] alamb opened a new pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
alamb opened a new pull request #9276:
URL: https://github.com/apache/arrow/pull/9276
The `create_batch_empty` function is used for creating output during aggregation. As part of my plan for better dictionary support it also needs to support DictionaryArray as well.
I think this may conflict with https://github.com/apache/arrow/pull/9114 from @ovr -- and if that PR will be worked on soon I think it should be merged prior to this one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-764615979
I agree that #9281 looks much better. Thanks @jorgecarleitao
I might port the tests from this PR subsently, but for now i don't think it is doing anything valuable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9276:
URL: https://github.com/apache/arrow/pull/9276
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-764615979
I agree that #9281 looks much better. Thanks @jorgecarleitao
I might port the tests from this PR subsently, but for now i don't think it is doing anything valuable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-764612354
fyi, with #9281 , (imo) this is solved more broadly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io edited a comment on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-763672876
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=h1) Report
> Merging [#9276](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=desc) (07d8ce8) into [master](https://codecov.io/gh/apache/arrow/commit/a0e12445cc8689befe40c20e2fac6e6df252bef6?el=desc) (a0e1244) will **increase** coverage by `0.01%`.
> The diff coverage is `80.41%`.
[](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9276 +/- ##
==========================================
+ Coverage 81.64% 81.66% +0.01%
==========================================
Files 215 215
Lines 52489 52512 +23
==========================================
+ Hits 42857 42884 +27
+ Misses 9632 9628 -4
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/src/physical\_plan/common.rs](https://codecov.io/gh/apache/arrow/pull/9276/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2NvbW1vbi5ycw==) | `82.03% <80.41%> (+7.74%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=footer). Last update [a0e1244...07d8ce8](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb closed pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
alamb closed pull request #9276:
URL: https://github.com/apache/arrow/pull/9276
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io edited a comment on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-763672876
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=h1) Report
> Merging [#9276](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=desc) (07d8ce8) into [master](https://codecov.io/gh/apache/arrow/commit/a0e12445cc8689befe40c20e2fac6e6df252bef6?el=desc) (a0e1244) will **increase** coverage by `0.01%`.
> The diff coverage is `80.41%`.
[](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9276 +/- ##
==========================================
+ Coverage 81.64% 81.66% +0.01%
==========================================
Files 215 215
Lines 52489 52512 +23
==========================================
+ Hits 42857 42884 +27
+ Misses 9632 9628 -4
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/src/physical\_plan/common.rs](https://codecov.io/gh/apache/arrow/pull/9276/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2NvbW1vbi5ycw==) | `82.03% <80.41%> (+7.74%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=footer). Last update [a0e1244...07d8ce8](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-763648633
https://issues.apache.org/jira/browse/ARROW-11327
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-764612354
fyi, with #9281 , (imo) this is solved more broadly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io commented on pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#issuecomment-763672876
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=h1) Report
> Merging [#9276](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=desc) (47bf88b) into [master](https://codecov.io/gh/apache/arrow/commit/691286975f277f00586cabc6d834ff1efd8caf8c?el=desc) (6912869) will **increase** coverage by `0.01%`.
> The diff coverage is `85.71%`.
[](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9276 +/- ##
==========================================
+ Coverage 81.68% 81.70% +0.01%
==========================================
Files 215 215
Lines 52561 52584 +23
==========================================
+ Hits 42935 42962 +27
+ Misses 9626 9622 -4
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/src/physical\_plan/common.rs](https://codecov.io/gh/apache/arrow/pull/9276/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2NvbW1vbi5ycw==) | `86.06% <85.71%> (+7.27%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=footer). Last update [6912869...47bf88b](https://codecov.io/gh/apache/arrow/pull/9276?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #9276: ARROW-11327: [Rust][DataFusion] Add DictionarySupport to create_batch_empty
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #9276:
URL: https://github.com/apache/arrow/pull/9276#discussion_r561003845
##########
File path: rust/datafusion/src/physical_plan/common.rs
##########
@@ -121,121 +128,130 @@ pub fn build_file_list(dir: &str, filenames: &mut Vec<String>, ext: &str) -> Res
Ok(())
}
-/// creates an empty record batch.
+/// Creates an empty (0 row) record batch with the specified schema
pub fn create_batch_empty(schema: &Schema) -> ArrowResult<RecordBatch> {
let columns = schema
.fields()
.iter()
- .map(|f| match f.data_type() {
Review comment:
this code is just hoisted into its own function so it can be called recursively
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org