You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "anjakefala (via GitHub)" <gi...@apache.org> on 2024/11/05 01:28:45 UTC
[I] [C++] Improve sequential access use of ChunkResolver [arrow]
anjakefala opened a new issue, #44641:
URL: https://github.com/apache/arrow/issues/44641
### Describe the enhancement requested
There has been recent work to move the [ChunkResolver](https://github.com/apache/arrow/issues/34535) to public API.
`ChunkResolver` uses `O(log(num_chunks))` binary search to identify chunks, which is optimised for random access. For sequential row-by-row access, using ChunkResolver would be inefficient.
Sometimes a user needs to be able to do row-major processing of the data. To that note, the proposal is to add these [helper methods](https://github.com/apache/arrow/issues/34535#issuecomment-1977304538) to the `ChunKResolver` API for more efficient sequential access traversal.
These helper methods were written by @felipecrv:
```
/// \pre loc.chunk_index >= 0
/// \pre loc.index_in_chunk is assumed valid if chunk_index is not the last one
inline bool Valid(ChunkLocation loc) const {
const int64_t last_chunk_index = static_cast<int64_t>(offsets_.size()) - 1;
return loc.chunk_index + 1 < last_chunk_index ||
(loc.chunk_index + 1 == last_chunk_index &&
loc.index_in_chunk < offsets_[last_chunk_index]);
}
/// \pre Valid(loc)
inline ChunkLocation Next(ChunkLocation loc) const {
const int64_t next_index_in_chunk = loc.index_in_chunk + 1;
return (next_index_in_chunk < offsets_[loc.chunk_index + 1])
? ChunkLocation{loc.chunk_index, next_index_in_chunk}
: ChunkLocation{loc.chunk_index + 1, 0};
}
```
with the resulting loops:
```
ChunkResolver resolver(batches);
for (ChunkLocation loc; resolver.Valid(loc); loc = resolved.Next(loc)) {
// re-use loc for all the typed columns since they are split on the same offsets
}
```
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #44641: [C++] Improve performance of sequential access of ChunkResolver
URL: https://github.com/apache/arrow/issues/44641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #44641:
URL: https://github.com/apache/arrow/issues/44641#issuecomment-2457294946
+1 I think these would be nice additions to the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #44641:
URL: https://github.com/apache/arrow/issues/44641#issuecomment-3921258585
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #44641:
URL: https://github.com/apache/arrow/issues/44641#issuecomment-3521359483
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #44641:
URL: https://github.com/apache/arrow/issues/44641#issuecomment-2461650145
I think this is a great idea, though we might need to either rename `ChunkResolver` or otherwise just ensure that the public docs make all the limitations clear.
That's really my only concern, we want to avoid this becoming the "standard way to iterate a table" and instead ensure that users can easily know when they should or shouldn't use the `ChunkResolver`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Improve performance of sequential access of ChunkResolver [arrow]
Posted by "anjakefala (via GitHub)" <gi...@apache.org>.
anjakefala commented on issue #44641:
URL: https://github.com/apache/arrow/issues/44641#issuecomment-2463449521
> instead ensure that users can easily know when they should or shouldn't use the ChunkResolver
This makes sense! So the steps for this PR are to add tests and appropriate documentation, on top of the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org