You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/11/11 23:36:36 UTC
[GitHub] [arrow] jorgecarleitao opened a new pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
jorgecarleitao opened a new pull request #8645:
URL: https://github.com/apache/arrow/pull/8645
This PR addresses 3 small issues on `MutableBuffer`:
1. `write_bytes` is incorrect, as it double-increments `len`: the length is incremented both on `self.write` and also by `write_bytes` itself. This leads to more allocations than necessary.
2. `write` is implemented from the trait `io::Write`. However, this trait is suitable for fallible IO operations. In the case of a write to memory, it isn't really fallible because we can always call `reserve` to allocate more space.
3. `write` and `write_bytes` are really similar.
This PR replaces both `write_bytes` and `write` by `extend_from_slice` (inspired by [`Vec::extend_from_slice`](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.extend_from_slice). This has the same performance (a single capacity comparison) or better. Compared to the current code, the main difference is that this is infallible as it `reserve` more space if the required size is insufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726749635
FWIW I ran the code in #8598 under valgrind and it does not appear to fix the issue https://github.com/apache/arrow/pull/8598#issuecomment-726749085
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#discussion_r522935301
##########
File path: rust/arrow/src/util/bit_util.rs
##########
@@ -99,36 +99,6 @@ pub unsafe fn unset_bit_raw(data: *mut u8, i: usize) {
*data.add(i >> 3) ^= BIT_MASK[i & 7];
}
-/// Sets bits in the non-inclusive range `start..end` for `data`
-///
-/// # Safety
-///
-/// Note this doesn't do any bound checking, for performance reason. The caller is
-/// responsible to guarantee that both `start` and `end` are within bounds.
-#[inline]
-pub unsafe fn set_bits_raw(data: *mut u8, start: usize, end: usize) {
Review comment:
🎉
##########
File path: rust/arrow/src/array/builder.rs
##########
@@ -525,7 +515,7 @@ impl<T: ArrowPrimitiveType> ArrayBuilder for PrimitiveBuilder<T> {
let sliced = array.buffers()[0].data();
// slice into data by factoring (offset and length) * byte width
self.values_builder
- .write_bytes(&sliced[(offset * mul)..((len + offset) * mul)], len)?;
+ .write_bytes(&sliced[(offset * mul)..((len + offset) * mul)], len);
Review comment:
Nice 👁️ -- I agree that removing the `len` parameter entirely would be the best course of action here
##########
File path: rust/arrow/src/array/builder.rs
##########
@@ -397,18 +396,9 @@ impl<T: ArrowPrimitiveType> BufferBuilder<T> {
/// Writes a byte slice to the underlying buffer and updates the `len`, i.e. the
/// number array elements in the builder. Also, converts the `io::Result`
/// required by the `Write` trait to the Arrow `Result` type.
- fn write_bytes(&mut self, bytes: &[u8], len_added: usize) -> Result<()> {
- let write_result = self.buffer.write(bytes);
- // `io::Result` has many options one of which we use, so pattern matching is
- // overkill here
- if write_result.is_err() {
- Err(ArrowError::MemoryError(
- "Could not write to Buffer, not big enough".to_string(),
- ))
- } else {
- self.len += len_added;
- Ok(())
- }
+ fn write_bytes(&mut self, bytes: &[u8], len_added: usize) {
Review comment:
I agree with @jhorstmann here that we should remove `len_added` (maybe as another PR) -- the length that is actually added is the `bytes.len()` rather than `len_added` so having the caller have to provide both leaves the opportunity for additional latent bugs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726510728
Ok, I confirm that this error is due to a wrong pointer offset on the `set_bits_raw` and that this undefined behavior is already present in `master`:
```
Switched to branch 'master'
Your branch is up to date with 'upstream/master'.
Compiling arrow v3.0.0-SNAPSHOT (/Users/jorgecarleitao/projects/arrow/rust/arrow)
Finished bench [optimized] target(s) in 53.79s
Running /Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c
Gnuplot not found, using plotters backend
bench_primitive time: [522.83 us 523.58 us 524.34 us]
thrpt: [7.4499 GiB/s 7.4607 GiB/s 7.4713 GiB/s]
change:
time: [-6.7844% -4.8815% -3.0463%] (p = 0.00 < 0.05)
thrpt: [+3.1420% +5.1320% +7.2782%]
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
3 (3.00%) high mild
6 (6.00%) high severe
Benchmarking bench_bool: Warming up for 3.0000 serror: process didn't exit successfully: `/Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c --bench` (signal: 11, SIGSEGV: invalid memory reference)
```
This PR now also removes that function altogether and replaces it by a safe counterpart. I suspect that there will be a performance regression, but I do not know enough of bit operations to be able to fix the function (and it is not clear whether we should, as per @vertexclique suggestion of using `bitvec`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726724609
I am going to take another hard look at https://github.com/apache/arrow/pull/8598 and see if we can get enough consensus to get it merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-725910876
@alamb @jhorstmann @vertexclique , I think I need your help here. This change is causing a segfault in one of the benchmarks, even though I have not changed any unsafe code and think the change is reasonable.
I narrowed it down to the function `set_bits_raw`. I think that this function is accessing a byte too many. However, my bit knowledge is limited here. I filled ARROW-10562 with why I think the function has an issue. IMO the reason this has not surfaced before is that we were allocating too much memory in the boolean buffers, and thus the memory address was fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vertexclique edited a comment on pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
vertexclique edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-725957761
My suggestion for this is merging https://github.com/apache/arrow/pull/8598 then implementing bit ops over that interface. And get rid of bit_util.rs. That's going to remove all these issues in a single shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726756578
Thanks a lot, @alamb , really useful data points ❤️
For me that is enough of a reason: fix UB with `safe` code, and figure out a way to perform multi-bit assignment on a single call fast on a subsequent PR.
I also agree with @jhorstmann about the `len_added`. I did not want to change the API for builders in this PR because this already does a backward incompatible change on `MutableBuffer` and one change like this per PR avoids mixing the discussions).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-725910876
@alamb @jhorstmann @vertexclique , I think I need your help here. This change is causing a segfault in one of the benchmarks, even though I have not changed any unsafe code and think the change is reasonable.
I narrowed it down to the function `set_bits_raw`. I think that this function is accessing a byte too many. However, my bit knowledge is limited here. I filled ARROW-10562 with why I think the function has an issue. IMO the reason this has not surfaced before is that we were allocating too much memory in the boolean buffers, and thus the memory address was fine.
(IMO we should just make that thing safe...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726510728
Ok, I confirm that this error is due to a wrong pointer offset on the `set_bits_raw` and that this undefined behavior is already present in `master`:
```
Switched to branch 'master'
Your branch is up to date with 'upstream/master'.
Compiling arrow v3.0.0-SNAPSHOT (/Users/jorgecarleitao/projects/arrow/rust/arrow)
Finished bench [optimized] target(s) in 53.79s
Running /Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c
Gnuplot not found, using plotters backend
bench_primitive time: [522.83 us 523.58 us 524.34 us]
thrpt: [7.4499 GiB/s 7.4607 GiB/s 7.4713 GiB/s]
change:
time: [-6.7844% -4.8815% -3.0463%] (p = 0.00 < 0.05)
thrpt: [+3.1420% +5.1320% +7.2782%]
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
3 (3.00%) high mild
6 (6.00%) high severe
Benchmarking bench_bool: Warming up for 3.0000 serror: process didn't exit successfully: `/Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c --bench` (signal: 11, SIGSEGV: invalid memory reference)
```
This PR removes now also removes that function altogether and replaces it by a safe counterpart. I suspect that there will be a performance regression, but I do not know enough of bit operations to be able to fix the function (and it is not clear whether we should, as per @vertexclique suggestion of using `bitvec`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb edited a comment on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726749635
FWIW I ran the code in #8598 under valgrind and it does not appear to fix the issue -- see details in https://github.com/apache/arrow/pull/8598#issuecomment-726749085
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] vertexclique commented on pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
vertexclique commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-725957761
My suggestion for this is merging https://github.com/apache/arrow/pull/8634 then implementing bit ops over that interface. And get rid of bit_util.rs. That's going to remove all these issues in a single shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726742048
> (if you be so kind, could you quickly run fd75933 , just to test whether this PR addresses the issue?)
@jorgecarleitao -- I did so. There are no errors reported by valgrind when I ran commit fd75933bead400a3943af1d336c006c6656d6f80 (HEAD, jorgecarleitao/buffer_write)
```
==7779== For lists of detected and suppressed errors, rerun with: -s
==7779== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726510728
Ok, I confirm that this error is due to a wrong byte offset on the `set_bits_raw` and that this undefined behavior is already present in `master`:
```
Switched to branch 'master'
Your branch is up to date with 'upstream/master'.
Compiling arrow v3.0.0-SNAPSHOT (/Users/jorgecarleitao/projects/arrow/rust/arrow)
Finished bench [optimized] target(s) in 53.79s
Running /Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c
Gnuplot not found, using plotters backend
bench_primitive time: [522.83 us 523.58 us 524.34 us]
thrpt: [7.4499 GiB/s 7.4607 GiB/s 7.4713 GiB/s]
change:
time: [-6.7844% -4.8815% -3.0463%] (p = 0.00 < 0.05)
thrpt: [+3.1420% +5.1320% +7.2782%]
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
3 (3.00%) high mild
6 (6.00%) high severe
Benchmarking bench_bool: Warming up for 3.0000 serror: process didn't exit successfully: `/Users/jorgecarleitao/projects/arrow/rust/target/release/deps/builder-65ece1c06584e62c --bench` (signal: 11, SIGSEGV: invalid memory reference)
```
This PR removes now also removes that function altogether and replaces it by a safe counterpart. I suspect that there will be a performance regression, but I do not know enough of bit operations to be able to fix the function (and it is not clear whether we should, as per @vertexclique suggestion of using `bitvec`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jhorstmann commented on a change in pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
jhorstmann commented on a change in pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#discussion_r522016134
##########
File path: rust/arrow/src/array/builder.rs
##########
@@ -525,7 +515,7 @@ impl<T: ArrowPrimitiveType> ArrayBuilder for PrimitiveBuilder<T> {
let sliced = array.buffers()[0].data();
// slice into data by factoring (offset and length) * byte width
self.values_builder
- .write_bytes(&sliced[(offset * mul)..((len + offset) * mul)], len)?;
+ .write_bytes(&sliced[(offset * mul)..((len + offset) * mul)], len);
Review comment:
I couldn't reproduce the issue yet, but this line looks a bit suspicious. The first parameter has a larger len (in bytes) than the second `len` parameter indicates (number of `T` elements).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jhorstmann commented on a change in pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
jhorstmann commented on a change in pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#discussion_r522001562
##########
File path: rust/arrow/src/array/builder.rs
##########
@@ -397,18 +396,9 @@ impl<T: ArrowPrimitiveType> BufferBuilder<T> {
/// Writes a byte slice to the underlying buffer and updates the `len`, i.e. the
/// number array elements in the builder. Also, converts the `io::Result`
/// required by the `Write` trait to the Arrow `Result` type.
- fn write_bytes(&mut self, bytes: &[u8], len_added: usize) -> Result<()> {
- let write_result = self.buffer.write(bytes);
- // `io::Result` has many options one of which we use, so pattern matching is
- // overkill here
- if write_result.is_err() {
- Err(ArrowError::MemoryError(
- "Could not write to Buffer, not big enough".to_string(),
- ))
- } else {
- self.len += len_added;
- Ok(())
- }
+ fn write_bytes(&mut self, bytes: &[u8], len_added: usize) {
Review comment:
@jorgecarleitao My guess would be that the issue is related to this `len_added` parameter. In the filter kernel this was used for additional padding, most other users probably interpreted this as the length of the bytes array. I would suggest removing this parameter, since you already implemented a workaround in the filter kernel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726736494
In terms of evidence that there is a problem on master, I ran the arrow test suite under `valgind` @ 30516049522c1a527ffb375e7790102f58edb4f9 on master and it does flag an invalid read (that could cause a segfault depending on the circumstances). Here is the error it flagged:
```
test compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8 ... ==5483== Invalid read of size 1
==5483== at 0x55D7B2: arrow::util::bit_util::set_bits_raw (bit_util.rs:128)
==5483== by 0x605D49: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::append_n (builder.rs:339)
==5483== by 0x62C854: arrow::array::builder::PrimitiveBuilder<T>::append_slice (builder.rs:591)
==5483== by 0xA992F6: arrow::array::builder::StringBuilder::append_value (builder.rs:1781)
==5483== by 0x6A309B: arrow::array::builder::StringDictionaryBuilder<K>::append (builder.rs:2435)
==5483== by 0x302DA3: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8 (cast.rs:2612)
==5483== by 0x31E499: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8::{{closure}} (cast.rs:2598)
==5483== by 0xA751ED: core::ops::function::FnOnce::call_once (function.rs:232)
==5483== by 0xB97365: call_once<(),FnOnce<()>> (boxed.rs:1008)
==5483== by 0xB97365: call_once<(),alloc::boxed::Box<FnOnce<()>>> (panic.rs:318)
==5483== by 0xB97365: do_call<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>,()> (panicking.rs:331)
==5483== by 0xB97365: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>> (panicking.rs:274)
==5483== by 0xB97365: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>,()> (panic.rs:394)
==5483== by 0xB97365: run_test_in_process (lib.rs:541)
==5483== by 0xB97365: test::run_test::run_test_inner::{{closure}} (lib.rs:450)
==5483== by 0xB969F8: test::run_test::run_test_inner (lib.rs:475)
==5483== by 0xB94BE9: test::run_test (lib.rs:505)
==5483== by 0xB829D8: run_tests<closure-2> (lib.rs:284)
==5483== by 0xB829D8: test::console::run_tests_console (console.rs:280)
==5483== Address 0x65c1900 is 0 bytes after a block of size 128 alloc'd
==5483== at 0x4C34443: memalign (vg_replace_malloc.c:906)
==5483== by 0x4C34546: posix_memalign (vg_replace_malloc.c:1070)
==5483== by 0xEB42F3: aligned_malloc (alloc.rs:95)
==5483== by 0xEB42F3: alloc (alloc.rs:22)
==5483== by 0xEB42F3: realloc_fallback (alloc.rs:39)
==5483== by 0xEB42F3: realloc (alloc.rs:50)
==5483== by 0xEB42F3: __rdl_realloc (alloc.rs:320)
==5483== by 0x55348C: alloc::alloc::realloc (alloc.rs:124)
==5483== by 0x71FB58: arrow::memory::reallocate (memory.rs:187)
==5483== by 0x9D5606: arrow::buffer::MutableBuffer::reserve (buffer.rs:665)
==5483== by 0x5FFC65: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::reserve (builder.rs:307)
==5483== by 0x605BEF: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::append_n (builder.rs:335)
==5483== by 0x62C854: arrow::array::builder::PrimitiveBuilder<T>::append_slice (builder.rs:591)
==5483== by 0xA992F6: arrow::array::builder::StringBuilder::append_value (builder.rs:1781)
==5483== by 0x6A309B: arrow::array::builder::StringDictionaryBuilder<K>::append (builder.rs:2435)
==5483== by 0x302DA3: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8 (cast.rs:2612)
==5483==
==5483== Invalid write of size 1
==5483== at 0x55D7B4: arrow::util::bit_util::set_bits_raw (bit_util.rs:128)
==5483== by 0x605D49: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::append_n (builder.rs:339)
==5483== by 0x62C854: arrow::array::builder::PrimitiveBuilder<T>::append_slice (builder.rs:591)
==5483== by 0xA992F6: arrow::array::builder::StringBuilder::append_value (builder.rs:1781)
==5483== by 0x6A309B: arrow::array::builder::StringDictionaryBuilder<K>::append (builder.rs:2435)
==5483== by 0x302DA3: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8 (cast.rs:2612)
==5483== by 0x31E499: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8::{{closure}} (cast.rs:2598)
==5483== by 0xA751ED: core::ops::function::FnOnce::call_once (function.rs:232)
==5483== by 0xB97365: call_once<(),FnOnce<()>> (boxed.rs:1008)
==5483== by 0xB97365: call_once<(),alloc::boxed::Box<FnOnce<()>>> (panic.rs:318)
==5483== by 0xB97365: do_call<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>,()> (panicking.rs:331)
==5483== by 0xB97365: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>> (panicking.rs:274)
==5483== by 0xB97365: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnOnce<()>>>,()> (panic.rs:394)
==5483== by 0xB97365: run_test_in_process (lib.rs:541)
==5483== by 0xB97365: test::run_test::run_test_inner::{{closure}} (lib.rs:450)
==5483== by 0xB969F8: test::run_test::run_test_inner (lib.rs:475)
==5483== by 0xB94BE9: test::run_test (lib.rs:505)
==5483== by 0xB829D8: run_tests<closure-2> (lib.rs:284)
==5483== by 0xB829D8: test::console::run_tests_console (console.rs:280)
==5483== Address 0x65c1900 is 0 bytes after a block of size 128 alloc'd
==5483== at 0x4C34443: memalign (vg_replace_malloc.c:906)
==5483== by 0x4C34546: posix_memalign (vg_replace_malloc.c:1070)
==5483== by 0xEB42F3: aligned_malloc (alloc.rs:95)
==5483== by 0xEB42F3: alloc (alloc.rs:22)
==5483== by 0xEB42F3: realloc_fallback (alloc.rs:39)
==5483== by 0xEB42F3: realloc (alloc.rs:50)
==5483== by 0xEB42F3: __rdl_realloc (alloc.rs:320)
==5483== by 0x55348C: alloc::alloc::realloc (alloc.rs:124)
==5483== by 0x71FB58: arrow::memory::reallocate (memory.rs:187)
==5483== by 0x9D5606: arrow::buffer::MutableBuffer::reserve (buffer.rs:665)
==5483== by 0x5FFC65: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::reserve (builder.rs:307)
==5483== by 0x605BEF: <arrow::array::builder::BufferBuilder<T> as arrow::array::builder::BufferBuilderTrait<T>>::append_n (builder.rs:335)
==5483== by 0x62C854: arrow::array::builder::PrimitiveBuilder<T>::append_slice (builder.rs:591)
==5483== by 0xA992F6: arrow::array::builder::StringBuilder::append_value (builder.rs:1781)
==5483== by 0x6A309B: arrow::array::builder::StringDictionaryBuilder<K>::append (builder.rs:2435)
==5483== by 0x302DA3: arrow::compute::kernels::cast::tests::test_cast_dict_to_dict_bad_index_value_utf8 (cast.rs:2612)
==5483==
```
My interpretation of this report is that `arrow::buffer::MutableBuffer::reserve` is reading off the end of the array. I haven't studied the code closely.
The actual command I used is below in case anyone is interested
PARQUET_TEST_DATA=`pwd`/../../cpp/submodules/parquet-testing/data ARROW_TEST_DATA=`pwd`/../../testing/data valgrind /home/andrew/Software/arrow/rust/target/debug/deps/arrow-b9bea680be7dc6e4
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726737738
(if you be so kind, could you quickly run fd75933 , just to test whether this PR addresses the issue?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-726756578
Thanks a lot, @alamb , really useful data points ❤️
For me that is enough of a reason: fix UB with `safe` code, and figure out a way to perform multi-bit assignment on a single call fast on a subsequent PR.
I also agree with @jhorstmann about the `len_added`. I did not want to change the API for builders in this PR because this already does a backward incompatible change on `MutableBuffer` and one change like this per PR avoids mixing the discussions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao closed pull request #8645: ARROW-10561: [Rust] Simplified Buffer's `write` and `write_bytes` and fixed undefined behavior
Posted by GitBox <gi...@apache.org>.
jorgecarleitao closed pull request #8645:
URL: https://github.com/apache/arrow/pull/8645
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8645: ARROW-10561: [Rust] Small simplification of `write` and `write_bytes`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8645:
URL: https://github.com/apache/arrow/pull/8645#issuecomment-725727007
https://issues.apache.org/jira/browse/ARROW-10561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org