You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/24 16:23:24 UTC
[GitHub] [arrow-rs] alamb opened a new issue #343: Add a RecordBatch::split
alamb opened a new issue #343:
URL: https://github.com/apache/arrow-rs/issues/343
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Sometimes it is advantageous to split one large `RecordBatch` into smaller batches for processing (for example, processing the multiple smaller `RecordBatch`es in parallel)
So instead of 1 `RecordBatch` with 1M rows, we could have 100 `RecordBatch`es with 10,000 rows each that could be processed in paralle.
@tustvold implemented such a function in https://github.com/apache/arrow-datafusion/pull/379/files
```
fn split_batch(sorted: &RecordBatch, batch_size: usize) -> Vec<RecordBatch> {
```
**Describe the solution you'd like**
Port the `split_batch` function into `RecordBatch::split(batch_size)` or something similar and add appropriate tests
cc @jorgecarleitao @nevi-me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] nevi-me commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
nevi-me commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-850938202
Reminder that slice currently doesn't work (or work correctly) for lists. So we have to be careful with how we use it.
It's sadly a limitation that @jorgecarleitao and I have encountered previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb closed issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
alamb closed issue #343:
URL: https://github.com/apache/arrow-rs/issues/343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-847219171
> We also have slice which already should make it quite easy to build this functionality (or use it to implement this functionality with).
Indeed -- I think that is how @tustvold implemented `split_batch`:
```rust
fn split_batch(sorted: &RecordBatch, batch_size: usize) -> Vec<RecordBatch> {
let batches = (sorted.num_rows() + batch_size - 1) / batch_size;
// Split the sorted RecordBatch into multiple
(0..batches)
.into_iter()
.map(|batch_idx| {
let columns = (0..sorted.num_columns())
.map(|column_idx| {
let length =
batch_size.min(sorted.num_rows() - batch_idx * batch_size);
sorted
.column(column_idx)
.slice(batch_idx * batch_size, length)
})
.collect();
RecordBatch::try_new(sorted.schema(), columns).unwrap()
})
.collect()
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-851388566
@nevi-me notes on #381
> we would need to account for its individual array offsets, as there is never a guarantee that a record batch has all child arrays starting from the same offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-879428117
Given we now have slice in https://github.com/apache/arrow-rs/issues/460 I don't think this adds much anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] Dandandan commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-847200279
We also have `slice` which already should make it quite easy to build this functionality (or use it to implement this functionality with).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org