You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/12/24 18:25:08 UTC
[GitHub] [arrow] andygrove opened a new pull request #9008: ARROW-11012: [Rust] [DataFusion] Make write_csv and write_parquet concurrent
andygrove opened a new pull request #9008:
URL: https://github.com/apache/arrow/pull/9008
Write partitions in parallel instead of one at a time. This results in a 4-5x speedup for the TPC-H file conversion process when repartitioning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] codecov-io commented on pull request #9008: ARROW-11012: [Rust] [DataFusion] Make write_csv and write_parquet concurrent
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #9008:
URL: https://github.com/apache/arrow/pull/9008#issuecomment-750943101
# [Codecov](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=h1) Report
> Merging [#9008](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=desc) (717a682) into [master](https://codecov.io/gh/apache/arrow/commit/2f5d59206f674ed1a1f5be6dbf809ba4d31b50da?el=desc) (2f5d592) will **increase** coverage by `0.33%`.
> The diff coverage is `73.68%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/9008/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #9008 +/- ##
==========================================
+ Coverage 82.54% 82.87% +0.33%
==========================================
Files 201 201
Lines 49983 49739 -244
==========================================
- Hits 41256 41220 -36
+ Misses 8727 8519 -208
```
| [Impacted Files](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [rust/datafusion/src/execution/context.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9leGVjdXRpb24vY29udGV4dC5ycw==) | `89.19% <73.68%> (-0.26%)` | :arrow_down: |
| [rust/parquet/src/encodings/encoding.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9wYXJxdWV0L3NyYy9lbmNvZGluZ3MvZW5jb2RpbmcucnM=) | `95.24% <0.00%> (-0.20%)` | :arrow_down: |
| [rust/arrow/src/array/transform/utils.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL3V0aWxzLnJz) | `100.00% <0.00%> (ø)` | |
| [rust/arrow/src/array/transform/primitive.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL3ByaW1pdGl2ZS5ycw==) | `100.00% <0.00%> (ø)` | |
| [rust/arrow/src/array/transform/variable\_size.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL3ZhcmlhYmxlX3NpemUucnM=) | `100.00% <0.00%> (ø)` | |
| [rust/arrow/src/util/bit\_chunk\_iterator.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvdXRpbC9iaXRfY2h1bmtfaXRlcmF0b3IucnM=) | `96.55% <0.00%> (+0.08%)` | :arrow_up: |
| [rust/arrow/src/array/transform/mod.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL21vZC5ycw==) | `88.20% <0.00%> (+1.01%)` | :arrow_up: |
| [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `83.87% <0.00%> (+2.05%)` | :arrow_up: |
| [rust/arrow/src/array/transform/fixed\_binary.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2ZpeGVkX2JpbmFyeS5ycw==) | `84.21% <0.00%> (+5.26%)` | :arrow_up: |
| [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/9008/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `98.47% <0.00%> (+36.88%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=footer). Last update [2f5d592...717a682](https://codecov.io/gh/apache/arrow/pull/9008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9008: ARROW-11012: [Rust] [DataFusion] Make write_csv and write_parquet concurrent
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9008:
URL: https://github.com/apache/arrow/pull/9008#issuecomment-750944847
https://issues.apache.org/jira/browse/ARROW-11012
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove closed pull request #9008: ARROW-11012: [Rust] [DataFusion] Make write_csv and write_parquet concurrent
Posted by GitBox <gi...@apache.org>.
andygrove closed pull request #9008:
URL: https://github.com/apache/arrow/pull/9008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org