You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/07/06 21:32:13 UTC

[GitHub] [arrow-rs] alamb opened a new issue, #4479: `RowCoverter::convert` that targets an existing `Rows`

alamb opened a new issue, #4479:
URL: https://github.com/apache/arrow-rs/issues/4479

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   I am trying to make group by really fast in DataFusion https://github.com/apache/arrow-datafusion/pull/6800
   
   
   The grouping code uses the Arrow Row format 👍  and calls `RowConverter::convert_columns`
   
   My traces imply that some non trivial amount of time is spent zeroing out newly allocated memory for `Rows`:
   
   ![Screenshot 2023-07-06 at 5 26 38 PM](https://github.com/apache/arrow-rs/assets/490673/00efc34f-491c-42d9-990c-22f920764410)
   
   
   
   **Describe the solution you'd like**
   
   I would like  a method like `RowConverter::convert_columns_in_place` that writes to an pre-existing rows (clearing it out first)
   
   Perhaps something like
   
   ```rust
   impl RowConverter {
   ...
       /// Convert [`ArrayRef`] columns into pre-existing [`Rows`], first calling `Rows::reset()` 
       ///
       /// See [`Row`] for information on when [`Row`] can be compared
       ///
       /// # Panics
       ///
       /// Panics if the schema of `columns` does not match that provided to [`RowConverter::new`]
       pub fn convert_columns_in_place(&mut self, columns: &[ArrayRef], dst &mut Rows) -> Result<ArrowError> {
   ```
   
   https://docs.rs/arrow-row/43.0.0/src/arrow_row/lib.rs.html#712
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features you've considered.
   -->
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] alamb commented on issue #4479: `RowCoverter::convert` that targets an existing `Rows`

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4479:
URL: https://github.com/apache/arrow-rs/issues/4479#issuecomment-1625380663

   I couldn't find anything else that does allocations per group, but I agree the profile is somewhat ambiguous


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] alamb commented on issue #4479: `RowCoverter::convert` that targets an existing `Rows`

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4479:
URL: https://github.com/apache/arrow-rs/issues/4479#issuecomment-1625381139

   (BTW this is like 2% of the total time, so more like a nice to have feature, rather than anything critical or really important)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold closed issue #4479: `RowCoverter::convert` that targets an existing `Rows`

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4479: `RowCoverter::convert` that targets an existing `Rows`
URL: https://github.com/apache/arrow-rs/issues/4479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on issue #4479: `RowCoverter::convert` that targets an existing `Rows`

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4479:
URL: https://github.com/apache/arrow-rs/issues/4479#issuecomment-1625300552

   I'm not very familiar with the Instruments profile, but it seems off to me that alloc_zeroed is under update_group_state and not convert_columns? If the allocation was occurring as part of row conversion I would have perhaps expected it to appear under convert_columns?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-rs] tustvold commented on issue #4479: `RowCoverter::convert` that targets an existing `Rows`

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4479:
URL: https://github.com/apache/arrow-rs/issues/4479#issuecomment-1657200585

   `label_issue.py` automatically added labels {'arrow'} from #4533


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org