You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/12/03 18:19:56 UTC
[GitHub] [arrow-rs] jhorstmann opened a new pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
jhorstmann opened a new pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998
# Which issue does this PR close?
Closes #997.
Draft for now since I still have to update the code comments and maybe add some more tests.
# Are there any user-facing changes?
I haven't run any benchmarks yet but this should improve performance of window functions with high-cardinality partition-by in datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998#issuecomment-985742856
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#998](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6f08306) into [master](https://codecov.io/gh/apache/arrow-rs/commit/e9be49d962560ce5b87544a2933d8b207322cf60?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e9be49d) will **decrease** coverage by `0.01%`.
> The diff coverage is `83.78%`.
> :exclamation: Current head 6f08306 differs from pull request most recent head 8899191. Consider uploading reports for the commit 8899191 to get more accurate results
[](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #998 +/- ##
==========================================
- Coverage 82.31% 82.30% -0.02%
==========================================
Files 168 168
Lines 48763 48790 +27
==========================================
+ Hits 40139 40156 +17
- Misses 8624 8634 +10
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow/src/compute/kernels/take.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUva2VybmVscy90YWtlLnJz) | `95.21% <ø> (-0.01%)` | :arrow_down: |
| [arrow/src/datatypes/field.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9maWVsZC5ycw==) | `53.37% <0.00%> (-0.31%)` | :arrow_down: |
| [...ng/src/flight\_server\_scenarios/auth\_basic\_proto.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW50ZWdyYXRpb24tdGVzdGluZy9zcmMvZmxpZ2h0X3NlcnZlcl9zY2VuYXJpb3MvYXV0aF9iYXNpY19wcm90by5ycw==) | `0.00% <ø> (ø)` | |
| [parquet/src/record/reader.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvcmVjb3JkL3JlYWRlci5ycw==) | `89.83% <0.00%> (-0.63%)` | :arrow_down: |
| [arrow/src/array/equal/utils.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2VxdWFsL3V0aWxzLnJz) | `74.00% <33.33%> (ø)` | |
| [arrow/src/compute/kernels/partition.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUva2VybmVscy9wYXJ0aXRpb24ucnM=) | `97.45% <97.36%> (-0.21%)` | :arrow_down: |
| [arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L3RyYW5zZm9ybS9ib29sZWFuLnJz) | `76.92% <100.00%> (ø)` | |
| [arrow/src/compute/util.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUvdXRpbC5ycw==) | `98.90% <100.00%> (ø)` | |
| [arrow/src/datatypes/schema.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9zY2hlbWEucnM=) | `66.66% <100.00%> (-0.27%)` | :arrow_down: |
| [arrow/src/ipc/writer.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2lwYy93cml0ZXIucnM=) | `84.12% <100.00%> (ø)` | |
| ... and [8 more](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e9be49d...8899191](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a change in pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998#discussion_r766036159
##########
File path: arrow/src/compute/kernels/partition.rs
##########
@@ -77,24 +74,51 @@ impl<'a> LexicographicalPartitionIterator<'a> {
/// see <https://en.wikipedia.org/wiki/Exponential_search>
#[inline]
fn exponential_search(
Review comment:
Can you please document what `start` and `end` and `bound` are? (specifically how the relate to each other -- I think as written it seems like `bound` is some starting index and the search starts at `start+bound` and stops at `end` indexes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jhorstmann commented on pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
jhorstmann commented on pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998#issuecomment-988064419
@Jimexist I'd like to get your feedback on this change, especially whether the comment in `LexicographicalPartitionIterator::next` sounds correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998#issuecomment-993947953
I'll plan to merge this PR in later this week (arrow 6.5 would get created at the end of next week, so I think we have plenty of time to land this one). Perhaps @Jimexist will have a chance to review by then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter edited a comment on pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998#issuecomment-985742856
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#998](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (caa05c7) into [master](https://codecov.io/gh/apache/arrow-rs/commit/e9be49d962560ce5b87544a2933d8b207322cf60?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e9be49d) will **increase** coverage by `0.00%`.
> The diff coverage is `97.36%`.
[](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #998 +/- ##
========================================
Coverage 82.31% 82.32%
========================================
Files 168 168
Lines 48763 49060 +297
========================================
+ Hits 40139 40388 +249
- Misses 8624 8672 +48
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow/src/compute/kernels/partition.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUva2VybmVscy9wYXJ0aXRpb24ucnM=) | `97.45% <97.36%> (-0.21%)` | :arrow_down: |
| [arrow/src/array/array\_string.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2FycmF5X3N0cmluZy5ycw==) | `97.08% <0.00%> (-0.83%)` | :arrow_down: |
| [parquet/src/record/reader.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvcmVjb3JkL3JlYWRlci5ycw==) | `89.83% <0.00%> (-0.63%)` | :arrow_down: |
| [parquet/src/schema/printer.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvc2NoZW1hL3ByaW50ZXIucnM=) | `72.47% <0.00%> (-0.55%)` | :arrow_down: |
| [arrow/src/datatypes/datatype.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9kYXRhdHlwZS5ycw==) | `65.95% <0.00%> (-0.43%)` | :arrow_down: |
| [arrow/src/datatypes/field.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9maWVsZC5ycw==) | `53.37% <0.00%> (-0.31%)` | :arrow_down: |
| [arrow/src/datatypes/schema.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2RhdGF0eXBlcy9zY2hlbWEucnM=) | `66.66% <0.00%> (-0.27%)` | :arrow_down: |
| [parquet/src/arrow/arrow\_array\_reader.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvYXJyb3cvYXJyb3dfYXJyYXlfcmVhZGVyLnJz) | `77.87% <0.00%> (-0.15%)` | :arrow_down: |
| [arrow/src/compute/kernels/take.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2NvbXB1dGUva2VybmVscy90YWtlLnJz) | `95.21% <0.00%> (-0.01%)` | :arrow_down: |
| [arrow/src/buffer/immutable.rs](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2J1ZmZlci9pbW11dGFibGUucnM=) | `97.84% <0.00%> (ø)` | |
| ... and [6 more](https://codecov.io/gh/apache/arrow-rs/pull/998/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e9be49d...caa05c7](https://codecov.io/gh/apache/arrow-rs/pull/998?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #998: Avoid allocating vector of indices in lexicographical_partition_ranges
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #998:
URL: https://github.com/apache/arrow-rs/pull/998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org