You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/19 12:00:41 UTC
[GitHub] [arrow-rs] alamb opened a new pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
alamb opened a new pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7
(Here is a PR to the new arrow crate to see what how this process is working!)
# Rationale / Usecase:
While writing tests (both in IOx and in DataFusion) where I need a single `RecordBatch`, I often find myself doing something like this (copied directly from IOx source code):
```rust
let schema = Arc::new(Schema::new(vec![
ArrowField::new("float_field", ArrowDataType::Float64, true),
ArrowField::new("time", ArrowDataType::Int64, true),
]));
let float_array: ArrayRef = Arc::new(Float64Array::from(vec![10.1, 20.1, 30.1, 40.1]));
let timestamp_array: ArrayRef = Arc::new(Int64Array::from(vec![1000, 2000, 3000, 4000]));
let batch = RecordBatch::try_new(schema, vec![float_array, timestamp_array])
.expect("created new record batch");
```
This is annoying because I have to redundantly (and verbosely) encode the information that `float_field` is a Float64 both in the `Schema` and the `Float64Array`
I would much rather be able to construct `RecordBatches` using a more Rust like style to avoid the the redundancy and reduce the amount of typing / redundancy:
# Proposed Change
As suggested in the comments from @returnString @nevi-me @jorgecarleitao in the draft PR: https://github.com/apache/arrow/pull/10063 add `try_from_iter` and `try_iter_with_null` functions
```rust
let record_batch = RecordBatch::try_from_iter(vec![
("a", Arc::new(a) as ArrayRef),
("b", Arc::new(b) as ArrayRef)
]).expect("valid conversion");
```
# TryFrom Implementation
Note I would really like to add a`TryFrom` implementation so I could write
```rust
let record_batch: RecordBatch = vec![
("a", Arc::new(a) as ArrayRef),
("b", Arc::new(b) as ArrayRef)
].try_into().expect("valid conversion");
```
However, when I tried to do so (with the following):
```rust
impl <I,F> TryFrom<I> for RecordBatch
where
I: IntoIterator<Item=(F, ArrayRef)>,
F: AsRef<str>,
{
type Error = ArrowError;
fn try_from(value: I) -> std::result::Result<Self, Self::Error> {
Self::try_from_iter(value)
}
}
```
I got the following compiler error
```
= note: conflicting implementation in crate `core`:
- impl<T, U> TryFrom<U> for T
where U: Into<T>;
```
Which appears to be a limitation of the Rust typesystem / compiler: See https://github.com/rust-lang/rust/issues/50133. Any help / suggestions from reviewers would be most appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-827167039
fyi @returnString
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-824174321
We had to perform a small re-write of master. The commits may look a bit odd, but it should not cause conflicts. Could you kindly rebase this against the latest master to make it easier to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-822417750
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eca9c6a) into [master](https://codecov.io/gh/apache/arrow-rs/commit/3f1380664251bf5dffdaed6daa6e0d6378e13ac7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3f13806) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head eca9c6a differs from pull request most recent head 5d3c337. Consider uploading reports for the commit 5d3c337 to get more accurate results
[](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7 +/- ##
==========================================
+ Coverage 82.47% 82.49% +0.01%
==========================================
Files 162 162
Lines 43414 43454 +40
==========================================
+ Hits 35805 35846 +41
+ Misses 7609 7608 -1
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL3JlY29yZF9iYXRjaC5ycw==) | `85.71% <100.00%> (+4.23%)` | :arrow_up: |
| [parquet/src/encodings/encoding.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvZW5jb2RpbmdzL2VuY29kaW5nLnJz) | `95.05% <0.00%> (+0.19%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3f13806...5d3c337](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-823153500
I rewrote master to remove the history of the other languages (and 50MB of history). This means this PR needs to be "rebased" against the current master.
The best way I found to do this was to find the relevant commits (via `git log`) and then cherry-pick them one at a time. Specifically, for this PR:
```
git reset --hard apache/master
git cherry-pick 5d3c337710b89871a46f6e623bad8c9e05419ed2
git push -f alamb
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] codecov-commenter edited a comment on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-822417750
# [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3933a28) into [master](https://codecov.io/gh/apache/arrow-rs/commit/3f1380664251bf5dffdaed6daa6e0d6378e13ac7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3f13806) will **increase** coverage by `0.01%`.
> The diff coverage is `81.34%`.
> :exclamation: Current head 3933a28 differs from pull request most recent head f235bec. Consider uploading reports for the commit f235bec to get more accurate results
[](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7 +/- ##
==========================================
+ Coverage 82.47% 82.49% +0.01%
==========================================
Files 162 162
Lines 43414 43454 +40
==========================================
+ Hits 35805 35846 +41
+ Misses 7609 7608 -1
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [arrow-flight/examples/server.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L2V4YW1wbGVzL3NlcnZlci5ycw==) | `0.00% <0.00%> (ø)` | |
| [arrow-flight/src/arrow.flight.protocol.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L3NyYy9hcnJvdy5mbGlnaHQucHJvdG9jb2wucnM=) | `0.00% <0.00%> (ø)` | |
| [arrow-flight/src/utils.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L3NyYy91dGlscy5ycw==) | `0.00% <0.00%> (ø)` | |
| [arrow-pyarrow-integration-testing/src/lib.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctcHlhcnJvdy1pbnRlZ3JhdGlvbi10ZXN0aW5nL3NyYy9saWIucnM=) | `0.00% <0.00%> (ø)` | |
| [arrow/src/alloc/types.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FsbG9jL3R5cGVzLnJz) | `0.00% <0.00%> (ø)` | |
| [arrow/src/array/builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIucnM=) | `85.29% <ø> (ø)` | |
| [arrow/src/array/iterator.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2l0ZXJhdG9yLnJz) | `95.80% <ø> (ø)` | |
| [arrow/src/array/null.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L251bGwucnM=) | `86.66% <ø> (ø)` | |
| [arrow/src/array/ord.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L29yZC5ycw==) | `59.28% <ø> (ø)` | |
| [arrow/src/array/raw\_pointer.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L3Jhd19wb2ludGVyLnJz) | `100.00% <ø> (ø)` | |
| ... and [94 more](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5b3298a...f235bec](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb merged pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-827301610
fyi, I think that this closes #210 , could you add it to the description so github picks it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org