You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "veluca93 (via GitHub)" <gi...@apache.org> on 2024/07/08 22:56:20 UTC
[PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
veluca93 opened a new pull request, #6025:
URL: https://github.com/apache/arrow-rs/pull/6025
This is a second step towards fixing https://github.com/apache/arrow-rs/issues/6020.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#issuecomment-2377787394
marking as draft until we can articulate the issue we are trying to solve (rather than just a source code analysis exercise)
Let me know what you think @veluca93
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1677155251
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
BTW this PR is still on my list to help clean, but I am struggling to find time.
Basically i plan to do some more code investigation to improve the comments here.
If there are any specific API changes / proposal to improve safety I think we should file a separate ticket to discuss them. I will do so if I find one during analysis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1688632493
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
Hi @veluca93 -- I wonder if you have had any more time to do some more code investigation? If you think this is actually unsound I will try and find more time to review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#issuecomment-2216941519
> FWIW #4261 may be related here, which attempted to move these to less unsafe APIs. IIRC it regressed performance significantly and there wasn't any user-facing unsoundness so I just abandoned it
I am not particularly concerned with the usage of the APIs being unsound (except in the cases where I explicitly mentioned it :-)), most of the `FIXME`s are just a "it is probably OK, but I can't follow this method well enough to figure out why and it's probably best for whoever wrote it to document why it is OK"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1700890057
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
Hi @alamb - any updates on when you might be able to dig a bit more into the safety invariants of this code? :-) unfortunately I don't think I am familiar enough with the codebase to effectively look more into it myself...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1677155607
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
Sounds good, take your time! I find with these things speediness should not trump thoroughness :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#issuecomment-2359319452
I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see https://github.com/apache/arrow-rs/issues/6418 for more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed pull request #6025: Unsafe improvements: arrow/{array_reader,buffer}.
URL: https://github.com/apache/arrow-rs/pull/6025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1673751655
##########
parquet/src/arrow/buffer/offset_buffer.rs:
##########
@@ -164,7 +168,8 @@ impl<I: OffsetSizeTrait> OffsetBuffer<I> {
let len = (end - start).to_usize().unwrap();
if len != 0 {
- // Safety: (1) the buffer is valid (2) the offsets are valid (3) the values in between are of ByteViewType
Review Comment:
Good to know - I rebased and got rid of the code!
##########
parquet/src/arrow/buffer/offset_buffer.rs:
##########
@@ -164,7 +168,8 @@ impl<I: OffsetSizeTrait> OffsetBuffer<I> {
let len = (end - start).to_usize().unwrap();
if len != 0 {
- // Safety: (1) the buffer is valid (2) the offsets are valid (3) the values in between are of ByteViewType
Review Comment:
Good to know - I rebased and got rid of the comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1673235443
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
I will review this more carefully -- I think this code is still sound as the only callers of `data_type` pass in the correct type.
However, maybe we could double check the data_type with an assert
##########
parquet/src/arrow/array_reader/struct_array.rs:
##########
@@ -173,6 +173,7 @@ impl ArrayReader for StructArrayReader {
array_data_builder.null_bit_buffer(Some(bitmap_builder.into()));
}
+ // SAFETY: FIXME: document why this call is safe.
Review Comment:
The argument basically goes something like the code in this module correctly constructed the struct array values and validated the input during the construction.
##########
parquet/src/arrow/buffer/offset_buffer.rs:
##########
@@ -164,7 +168,8 @@ impl<I: OffsetSizeTrait> OffsetBuffer<I> {
let len = (end - start).to_usize().unwrap();
if len != 0 {
- // Safety: (1) the buffer is valid (2) the offsets are valid (3) the values in between are of ByteViewType
Review Comment:
I think this code is slated for removal by @XiangpengHao in https://github.com/apache/arrow-rs/pull/6040, FWIW
##########
parquet/src/arrow/array_reader/primitive_array.rs:
##########
@@ -179,6 +179,11 @@ where
.add_buffer(record_data)
.null_bit_buffer(self.record_reader.consume_bitmap_buffer());
+ // SAFETY: Assuming that the RecordReader's null buffer is sufficiently sized for the
Review Comment:
I think the safety argument goes something like the RecordReader writes only valid values for type `T` into `record_data` and that the record_reader's null buffer was correctly created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1673746815
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
Checking data_type SGTM.
Technically speaking, relying on callers doing "the right thing" is _unsound_ (which is not the same as "exhibits undefined behaviour): unsafe code is sound if _no safe code could cause UB_, regardless of current usage.
Of course, if this is not exposed to the user, it is less important to fix, but would still be nice IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1673746815
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
Checking data_type SGTM.
Technically speaking, relying on callers doing "the right thing" is _unsound_ (which is not the same as "exhibits undefined behaviour"): unsafe code is sound if _no safe code could cause UB_, regardless of current usage.
Of course, if this is not exposed to the user, it is less important to fix, but would still be nice IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1689160729
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
I haven't had a lot of time - I've been busy and I am at a programming competition this week - but I can confirm that the code as written is unsound (even if the unsoundness cannot trigger undefined behaviour): validating the data type, or marking the function as unsafe and writing down in the caller why the data type is correct would fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#issuecomment-2216923637
FWIW https://github.com/apache/arrow-rs/pull/4261 may be related here, which attempted to move these to less unsafe APIs. IIRC it regressed performance significantly and there wasn't any user-facing unsoundness so I just abandoned it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1700896255
##########
parquet/src/arrow/buffer/dictionary_buffer.rs:
##########
@@ -162,6 +162,10 @@ impl<K: ArrowNativeType + Ord, V: OffsetSizeTrait> DictionaryBuffer<K, V> {
let data = match cfg!(debug_assertions) {
true => builder.build().unwrap(),
+ // SAFETY: FIXME: this is unsound. data_type is passed by the caller without
Review Comment:
I am likewise not familar with this part of the code -- and I haven't been able to find time to spend studying it in detail.
One of the nice things about open source code is that it is all available for everyone to look -- I welcome additional thoughts / reviews / assistance on this item
FRom my perspective, this ticket is current a reasearch item as it is a seemingly theoretical problem. Thus it will likely will remain a relatively low priority for me personally unless someone can cook up a rust example showing actual unsafe/unsound behavior resulting
thank you again for the help. It would be great if you could find time to analyze the code or make an example showing a problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Unsafe improvements: arrow/{array_reader,buffer}. [arrow-rs]
Posted by "veluca93 (via GitHub)" <gi...@apache.org>.
veluca93 commented on code in PR #6025:
URL: https://github.com/apache/arrow-rs/pull/6025#discussion_r1673752937
##########
parquet/src/arrow/array_reader/struct_array.rs:
##########
@@ -173,6 +173,7 @@ impl ArrayReader for StructArrayReader {
array_data_builder.null_bit_buffer(Some(bitmap_builder.into()));
}
+ // SAFETY: FIXME: document why this call is safe.
Review Comment:
Yep, that's what I assumed, but the method was a bit too long and I am not familiar enough with the library to really figure out the details here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org