[GitHub] [arrow-rs] alamb commented on issue #343: Add a RecordBatch::split to split large batches into a set of smaller batches

[GitBox <gi...@apache.org>]

alamb commented on issue #343:
URL: https://github.com/apache/arrow-rs/issues/343#issuecomment-847219171


   > We also have slice which already should make it quite easy to build this functionality (or use it to implement this functionality with).
   
   Indeed -- I think that is how @tustvold implemented `split_batch`:
   
   ```rust
      fn split_batch(sorted: &RecordBatch, batch_size: usize) -> Vec<RecordBatch> {
           let batches = (sorted.num_rows() + batch_size - 1) / batch_size;
   
           // Split the sorted RecordBatch into multiple
           (0..batches)
               .into_iter()
               .map(|batch_idx| {
                   let columns = (0..sorted.num_columns())
                       .map(|column_idx| {
                           let length =
                               batch_size.min(sorted.num_rows() - batch_idx * batch_size);
   
                           sorted
                               .column(column_idx)
                               .slice(batch_idx * batch_size, length)
                       })
                       .collect();
   
                   RecordBatch::try_new(sorted.schema(), columns).unwrap()
               })
               .collect()
       }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org