You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/19 12:00:41 UTC

[GitHub] [arrow-rs] alamb opened a new pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

alamb opened a new pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7


   (Here is a PR to the new arrow crate to see what how this process is working!)
   
   # Rationale / Usecase:
   
   While writing tests (both in IOx and in DataFusion) where I need a single `RecordBatch`, I often find myself doing something like this (copied directly from IOx source code):
   
   ```rust
   let schema = Arc::new(Schema::new(vec![
       ArrowField::new("float_field", ArrowDataType::Float64, true),
       ArrowField::new("time", ArrowDataType::Int64, true),
   ]));
   
   let float_array: ArrayRef = Arc::new(Float64Array::from(vec![10.1, 20.1, 30.1, 40.1]));
   let timestamp_array: ArrayRef = Arc::new(Int64Array::from(vec![1000, 2000, 3000, 4000]));
   
   let batch = RecordBatch::try_new(schema, vec![float_array, timestamp_array])
       .expect("created new record batch");
   ```
   
   This is annoying because I have to redundantly (and verbosely) encode the information that `float_field` is a Float64 both in the `Schema` and the `Float64Array`
   
   I would much rather  be able to construct `RecordBatches` using a more Rust like style to avoid the the redundancy and reduce the amount of typing / redundancy:
   
   # Proposed Change
   As suggested in the comments from @returnString @nevi-me @jorgecarleitao in the draft PR: https://github.com/apache/arrow/pull/10063 add `try_from_iter` and `try_iter_with_null` functions
   
   ```rust
   let record_batch = RecordBatch::try_from_iter(vec![
     ("a", Arc::new(a) as ArrayRef),
     ("b", Arc::new(b) as ArrayRef)
   ]).expect("valid conversion");
   ```
   
   # TryFrom Implementation
   
   Note I would really like to add a`TryFrom` implementation so I could write
   
   ```rust
   let record_batch: RecordBatch = vec![
     ("a", Arc::new(a) as ArrayRef),
     ("b", Arc::new(b) as ArrayRef)
   ].try_into().expect("valid conversion");
   ```
   
   However, when I tried to do so (with the following):
   
   ```rust
   impl <I,F> TryFrom<I> for RecordBatch
   where
        I: IntoIterator<Item=(F, ArrayRef)>,
        F: AsRef<str>,
   {
       type Error = ArrowError;
   
       fn try_from(value: I) -> std::result::Result<Self, Self::Error> {
           Self::try_from_iter(value)
       }
   }
   ```
   
   I got the following compiler error
   
   ```
        = note: conflicting implementation in crate `core`:
               - impl<T, U> TryFrom<U> for T
                 where U: Into<T>;
   
   ```
   
   Which appears to be a limitation of the Rust typesystem / compiler: See https://github.com/rust-lang/rust/issues/50133. Any help / suggestions from reviewers would be most appreciated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-827167039


   fyi @returnString 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] jorgecarleitao commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-824174321


   We had to perform a small re-write of master. The commits may look a bit odd, but it should not cause conflicts. Could you kindly rebase this against the latest master to make it easier to review?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] codecov-commenter commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-822417750


   # [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eca9c6a) into [master](https://codecov.io/gh/apache/arrow-rs/commit/3f1380664251bf5dffdaed6daa6e0d6378e13ac7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3f13806) will **increase** coverage by `0.01%`.
   > The diff coverage is `100.00%`.
   
   > :exclamation: Current head eca9c6a differs from pull request most recent head 5d3c337. Consider uploading reports for the commit 5d3c337 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow-rs/pull/7/graphs/tree.svg?width=650&height=150&src=pr&token=pq9V9qWZ1N&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master       #7      +/-   ##
   ==========================================
   + Coverage   82.47%   82.49%   +0.01%     
   ==========================================
     Files         162      162              
     Lines       43414    43454      +40     
   ==========================================
   + Hits        35805    35846      +41     
   + Misses       7609     7608       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL3JlY29yZF9iYXRjaC5ycw==) | `85.71% <100.00%> (+4.23%)` | :arrow_up: |
   | [parquet/src/encodings/encoding.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGFycXVldC9zcmMvZW5jb2RpbmdzL2VuY29kaW5nLnJz) | `95.05% <0.00%> (+0.19%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3f13806...5d3c337](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-823153500


   I rewrote master to remove the history of the other languages (and 50MB of history). This means this PR needs to be "rebased" against the current master.
   
   The best way I found to do this was to find the relevant commits (via `git log`) and then cherry-pick them one at a time. Specifically, for this PR:
   ```
   git reset --hard apache/master
   git cherry-pick 5d3c337710b89871a46f6e623bad8c9e05419ed2
   git push -f alamb
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] codecov-commenter edited a comment on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-822417750


   # [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3933a28) into [master](https://codecov.io/gh/apache/arrow-rs/commit/3f1380664251bf5dffdaed6daa6e0d6378e13ac7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3f13806) will **increase** coverage by `0.01%`.
   > The diff coverage is `81.34%`.
   
   > :exclamation: Current head 3933a28 differs from pull request most recent head f235bec. Consider uploading reports for the commit f235bec to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow-rs/pull/7/graphs/tree.svg?width=650&height=150&src=pr&token=pq9V9qWZ1N&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master       #7      +/-   ##
   ==========================================
   + Coverage   82.47%   82.49%   +0.01%     
   ==========================================
     Files         162      162              
     Lines       43414    43454      +40     
   ==========================================
   + Hits        35805    35846      +41     
   + Misses       7609     7608       -1     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [arrow-flight/examples/server.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L2V4YW1wbGVzL3NlcnZlci5ycw==) | `0.00% <0.00%> (ø)` | |
   | [arrow-flight/src/arrow.flight.protocol.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L3NyYy9hcnJvdy5mbGlnaHQucHJvdG9jb2wucnM=) | `0.00% <0.00%> (ø)` | |
   | [arrow-flight/src/utils.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctZmxpZ2h0L3NyYy91dGlscy5ycw==) | `0.00% <0.00%> (ø)` | |
   | [arrow-pyarrow-integration-testing/src/lib.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3ctcHlhcnJvdy1pbnRlZ3JhdGlvbi10ZXN0aW5nL3NyYy9saWIucnM=) | `0.00% <0.00%> (ø)` | |
   | [arrow/src/alloc/types.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FsbG9jL3R5cGVzLnJz) | `0.00% <0.00%> (ø)` | |
   | [arrow/src/array/builder.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2J1aWxkZXIucnM=) | `85.29% <ø> (ø)` | |
   | [arrow/src/array/iterator.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L2l0ZXJhdG9yLnJz) | `95.80% <ø> (ø)` | |
   | [arrow/src/array/null.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L251bGwucnM=) | `86.66% <ø> (ø)` | |
   | [arrow/src/array/ord.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L29yZC5ycw==) | `59.28% <ø> (ø)` | |
   | [arrow/src/array/raw\_pointer.rs](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-YXJyb3cvc3JjL2FycmF5L3Jhd19wb2ludGVyLnJz) | `100.00% <ø> (ø)` | |
   | ... and [94 more](https://codecov.io/gh/apache/arrow-rs/pull/7/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [5b3298a...f235bec](https://codecov.io/gh/apache/arrow-rs/pull/7?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] alamb merged pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
alamb merged pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-rs] jorgecarleitao commented on pull request #7: ARROW-12411: [Rust] Create RecordBatches from Iterators

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7:
URL: https://github.com/apache/arrow-rs/pull/7#issuecomment-827301610


   fyi, I think that this closes #210 , could you add it to the description so github picks it up?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org