[I] `arrow::util` test/bench APIs leak the `rand` version, breaking downstreams on a different rand major [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb opened a new issue, #10685:
URL: https://github.com/apache/arrow-rs/issues/10685

   **Is your feature request related to a problem or challenge?**
   - Found while testing upgrade of arrow in DataFusion: https://github.com/apache/datafusion/pull/24366
   - Related to https://github.com/apache/arrow-rs/pull/10566
   
   While updating DataFusion to arrow 60 I found that the `rand` `0.10` upgrade in https://github.com/apache/arrow-rs/pull/10566 breaks downstream users of `arrow::util::bench_util` and `arrow::util::test_util` that are on a different `rand` major. These APIs expose arrow's `rand` version in their signatures:
   
   - `bench_util::create_primitive_array` has a `StandardUniform: Distribution<T::Native>` bound, which callers must restate in their own generic code — but they can only name *their* `rand`'s `StandardUniform`
   - `test_util::seedable_rng` returns arrow's `rand`'s `StdRng`
   
   DataFusion is stuck on `rand` `0.9` for now (`rand_distr` `0.5` has no rand-0.10-compatible release), so generic bench helpers like this no longer compile:
   
   ```rust
   use rand::distr::{Distribution, StandardUniform}; // rand 0.9
   
   pub fn create_list_array<T>(size: usize, null_density: f32) -> ListArray
   where
       T: ArrowPrimitiveType,
       StandardUniform: Distribution<T::Native>, // rand 0.9 bound: cannot satisfy
                                                 // create_primitive_array's rand 0.10 bound
   {
       let values = create_primitive_array::<T>(length, 0.0);
       // ...
   }
   ```
   
   We had to remove the generic bound (pinning `T::Native = i64`) and replace `seedable_rng` with a local copy:
   
   ```rust
   /// Returns a fixed-seed RNG using this crate's `rand` version (arrow's
   /// `test_util::seedable_rng` returns its own `rand` version's `StdRng`)
   fn seedable_rng() -> StdRng {
       StdRng::seed_from_u64(42)
   }
   ```
   
   **Describe the solution you'd like**
   
   Re-export arrow's `rand` (e.g. `pub use rand;` from `arrow::util` under the relevant feature) so downstreams can name the right `StandardUniform`/`StdRng` types when calling these APIs, regardless of which `rand` they depend on themselves.
   
   At a minimum, call this out in the arrow 60 release notes / upgrade guide, since the breakage only shows up in downstream test and bench code.
   
   **Describe alternatives you've considered**
   
   Downstreams can avoid the generic bounds and copy `seedable_rng` locally, as shown above.
   
   **Additional context**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] re-export `arrow::util` `rand` version, breaking downstreams on a different rand major [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #10685: re-export `arrow::util` `rand` version, breaking downstreams on a different rand major
URL: https://github.com/apache/arrow-rs/issues/10685


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org