You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Abhisheklearn12 (via GitHub)" <gi...@apache.org> on 2026/07/25 20:37:37 UTC
[PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Abhisheklearn12 opened a new pull request, #10436:
URL: https://github.com/apache/arrow-rs/pull/10436
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Closes #8985
Supersedes #9768, which I closed it implemented these fast paths ungated, and benchmarking showed that made things significantly slower. Details below.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
`dictionary_cast` can build a view array from a dictionary two ways:
- (a) `unpack_dictionary` builds one view per dictionary *value*, then gathers with `take`
- (b) `view_from_dict_values` builds one view per *row* directly against the values buffer
The comment above (b) says (a) "incurs unnecessary data copy of the value buffer". That
isn't the case. `impl From<&GenericByteArray> for GenericByteViewArray` reuses the buffer
(it only falls back to copying when the values exceed the `u32` offset a view can hold),
and `take_byte_view` passes `data_buffers().to_vec()` through untouched. Neither step
copies value data, so there was no copy to remove.
What the two actually trade is a bandwidth-bound `u128` gather in `take` against a
per-row `make_view` that reads each row's payload out of the values buffer. The gather
wins once rows reach roughly 0.6x the dictionary size so (b) is a large pessimisation in
the common case, and only pays off when the dictionary is substantially larger than the
array is long, where (a) spends most of its time building views no row references.
This means the existing `Utf8 -> Utf8View` and `Binary -> BinaryView` fast paths on main
are currently **5-8x slower** than `unpack_dictionary` at typical batch shapes.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Gate the direct path on `keys.len() < values.len() / 2`; everything else falls through
to `unpack_dictionary`. The measured crossover is near `rows ≈ 0.6 * values`; the gate
switches short of it, at `rows = values / 2`, where the direct path is still ahead by
15-30%. That margin covers the crossover moving with cache size or microarchitecture.
Being wrong in that direction only forgoes a win; being wrong the other way is a
regression on the common case.
- Extend the direct path to the remaining combinations #8985 asks for:
`LargeUtf8`/`LargeBinary` -> `Utf8View`/`BinaryView`, and the `Utf8` <-> `Binary` cross
casts with the offset-fit check and UTF-8 validation the issue calls out.
- Fix `view_from_dict_values` dropping null dictionary *values*: they became empty strings
rather than nulls, where `unpack_dictionary` and `impl From<&GenericByteArray>` both
produce nulls.
- Bounds-check the dictionary key before indexing the offsets, turning an out-of-range key
from undefined behaviour into an error. The gate confines this loop to short row counts,
so it is within noise when the dictionary holds no nulls. When the dictionary does hold
nulls, the null check costs 25-40% on the direct path an unavoidable random bitmap
lookup per row, and the price of emitting nulls rather than the empty strings the
previous code produced.
## Benchmarks
Existing fast paths on main:
| cast | rows | dict values | main | this PR | change |
|---|---|---|---|---|---|
| `Utf8->Utf8View` | 8,192 | 100 | 26.3 µs | 3.4 µs | **-87.2%** |
| `Utf8->Utf8View` | 1,000,000 | 1,000 | 3188.9 µs | 554.6 µs | **-82.6%** |
| `Binary->BinaryView` | 8,192 | 100 | 24.9 µs | 3.3 µs | **-86.6%** |
| `Binary->BinaryView` | 1,000,000 | 1,000 | 3296.0 µs | 593.6 µs | **-82.0%** |
New arms, sparse shapes (dictionary larger than the array):
| cast | rows | dict values | main | this PR | change |
|---|---|---|---|---|---|
| `LargeUtf8->Utf8View` | 1,000 | 100,000 | 271.2 µs | 3.9 µs | **-98.6%** |
| `LargeUtf8->BinaryView` | 10,000 | 1,000,000 | 15752.5 µs | 78.6 µs | **-99.5%** |
| `LargeBinary->BinaryView` | 1,000 | 100,000 | 276.1 µs | 3.9 µs | **-98.6%** |
| `Utf8->BinaryView` | 10,000 | 1,000,000 | 3005.8 µs | 73.1 µs | **-97.6%** |
| `Binary->Utf8View` | 1,000 | 100,000 | 351.5 µs | 88.6 µs | **-74.8%** |
| `LargeBinary->Utf8View` | 10,000 | 1,000,000 | 4570.7 µs | 1713.9 µs | **-62.5%** |
(`Binary`/`LargeBinary -> Utf8View` gain less because UTF-8 validation of the dictionary
values dominates.)
Dense shapes on the new arms are unchanged, -2.9% to +1.2% the gate routes them to
`unpack_dictionary`. Worst cell across the whole matrix is +5.0%, on a shape where both
versions take the direct path and the delta is the null and bounds checks above.
<details>
<summary>Method</summary>
Both implementations were compiled into a single binary and timed in alternating rounds so
thermal drift cancels out of the ratio. Cells whose code is identical in both versions come
out at +0.0% to +1.2%, which bounds the method's noise; a two-run criterion comparison of
those same cells reported up to +47% drift, which is why it wasn't used. Outputs are
asserted equal per cell before timing. Measured on an i7-11700F; a second independent run
reproduced every headline result within 3.2 points and produced the identical set of cells
above 50%; the largest shift on any cell was 8.3 points, on a cell that is flat in both runs.
</details>
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
Yes. Every arm is exercised through **both** implementations (row counts either side of the
gate), covering null dictionary values, null keys, invalid UTF-8 under both `safe` and
strict `CastOptions`. Results are asserted equal to the
`unpack_dictionary` reference in each case.
# Are there any user-facing changes?
Casting `Dictionary<_, Utf8> -> Utf8View` and `Dictionary<_, Binary> -> BinaryView` becomes
substantially faster at typical batch shapes 5-8x on the shapes benchmarked above.
`Dictionary<_, LargeUtf8/LargeBinary> -> Utf8View/BinaryView` and the `Utf8` <-> `Binary`
cross casts gain a fast path when the dictionary holds more than twice as many values as the
array has rows. Outside that they behave as before, going through `unpack_dictionary`.
Two behaviour changes, called out explicitly:
1. A null dictionary *value* now casts to null rather than an empty string:
```rust
// values ["aa", NULL, "cc"], keys [0, 1, 2]
cast(&dict, &DataType::Utf8View) // before: ["aa", "", "cc"]
// after: ["aa", null, "cc"]
```
2. An out-of-range dictionary key now returns an InvalidArgumentError instead of being
undefined behaviour. Only reachable for a dictionary built without validation.
Happy to split either into its own PR if you'd prefer this one stay purely about the fast path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234371550
run benchmark cast_kernels
env:
BENCH_FILTER: dict.*to.*view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234390075
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234371550)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix/dict-view-cardinality-gate (93a5cb685f0841afd263359c296a1ff6069eb9e3) to bb82f3e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/bb82f3e4fcb99cea6c730e906c067579802593fe..93a5cb685f0841afd263359c296a1ff6069eb9e3)
<details><summary>Run configuration</summary>
```yaml
run benchmark cast_kernels
env:
BENCH_FILTER: "dict.*to.*view"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group fix_dict-view-cardinality-gate main
----- ------------------------------ ----
cast binary dict to string view (sparse) 1.00 57.4±1.93µs ? ?/sec 2.78 159.7±10.12µs ? ?/sec
cast dict to string view 1.00 16.5±0.01µs ? ?/sec 2.81 46.2±0.57µs ? ?/sec
cast dict to string view (sparse) 1.01 5.3±0.06µs ? ?/sec 1.00 5.2±0.07µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 35.0s |
| Peak memory | 9.9 MiB |
| Avg memory | 7.7 MiB |
| CPU user | 28.8s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 35.0s |
| Peak memory | 9.8 MiB |
| Avg memory | 9.0 MiB |
| CPU user | 32.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5125752661
> Happy to split either into its own PR if you'd prefer this one stay purely about the fast path.
could we pull out the bug fix changes into a separate PR from the performance improvements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234411759
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234396693)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix/dict-view-cardinality-gate (93a5cb685f0841afd263359c296a1ff6069eb9e3) to bb82f3e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/bb82f3e4fcb99cea6c730e906c067579802593fe..93a5cb685f0841afd263359c296a1ff6069eb9e3)
<details><summary>Run configuration</summary>
```yaml
run benchmark cast_kernels
env:
BENCH_FILTER: "dict.*to.*view"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group fix_dict-view-cardinality-gate main
----- ------------------------------ ----
cast binary dict to string view (sparse) 1.00 44.7±0.11µs ? ?/sec 3.56 159.2±10.66µs ? ?/sec
cast dict to string view 1.00 16.5±0.01µs ? ?/sec 2.80 46.2±0.56µs ? ?/sec
cast dict to string view (sparse) 1.02 5.3±0.07µs ? ?/sec 1.00 5.2±0.07µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 35.0s |
| Peak memory | 13.6 MiB |
| Avg memory | 8.1 MiB |
| CPU user | 29.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 35.0s |
| Peak memory | 13.1 MiB |
| Avg memory | 8.8 MiB |
| CPU user | 31.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5152899585
done @Jefffrey, split out. the null handling and bounds check (bug fixes) are in #10510 now, this one is perf only.
also imo, probably worth merging #10510 first, since this PR extends the direct path to more type combos and would spread the null bug until the fix lands, wdyt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3794229957
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
Review Comment:
ahhh you're right, i was only thinking about varying the key count. padding the values does it much more cleanly, thanks you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3791178687
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
Review Comment:
this test is really confusing; its named "both_paths_agree" but we're not testing them with the same input data. so we're not testing them against each as the name suggests, just in parallel? can we clarify what we're testing here
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
Review Comment:
we could simplify this setup by casting the input `utf8` to the required binary/largebinary etc. type
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
+ (
+ "LargeBinary->BinaryView",
+ large_binary,
+ DataType::BinaryView,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_direct),
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_unpack),
+ "{label} (unpack path)"
+ );
+ }
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_both_paths() {
+ // Invalid UTF-8 must behave identically whichever implementation runs, for both
+ // Binary and LargeBinary sources.
+ let mut b32 = BinaryBuilder::new();
+ let mut b64 = GenericBinaryBuilder::<i64>::new();
+ for v in [b"aa".as_slice(), b"bb", &[0xFF, 0xFE], b"dd", b"ee", b"ff"] {
+ b32.append_value(v);
+ b64.append_value(v);
+ }
+ let binary: ArrayRef = Arc::new(b32.finish());
+ let large_binary: ArrayRef = Arc::new(b64.finish());
Review Comment:
feels like we can greatly simplify this by using [`from_vec`](https://docs.rs/arrow/latest/arrow/array/type.BinaryArray.html#method.from_vec) for both types
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
+ (
+ "LargeBinary->BinaryView",
+ large_binary,
+ DataType::BinaryView,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_direct),
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_unpack),
+ "{label} (unpack path)"
+ );
+ }
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_both_paths() {
+ // Invalid UTF-8 must behave identically whichever implementation runs, for both
+ // Binary and LargeBinary sources.
+ let mut b32 = BinaryBuilder::new();
+ let mut b64 = GenericBinaryBuilder::<i64>::new();
+ for v in [b"aa".as_slice(), b"bb", &[0xFF, 0xFE], b"dd", b"ee", b"ff"] {
+ b32.append_value(v);
+ b64.append_value(v);
+ }
+ let binary: ArrayRef = Arc::new(b32.finish());
+ let large_binary: ArrayRef = Arc::new(b64.finish());
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let safe = CastOptions {
+ safe: true,
+ ..Default::default()
+ };
+
+ for values in [binary, large_binary] {
+ for keys in [keys_taking_direct_path(), keys_taking_unpack_path()] {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values.clone()).unwrap();
+
+ let err = cast_with_options(&dict, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+
+ let casted = cast_with_options(&dict, &DataType::Utf8View, &safe).unwrap();
+ let got: Vec<_> = casted.as_string_view().iter().collect();
+ // only rows whose key points at the invalid value are nullified
+ assert!(got.iter().all(|v| *v != Some("\u{FFFD}")));
Review Comment:
this is confusing; we're not checking the nullability we're checking the exact value? so the comment is misleading
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
Review Comment:
i feel we can unify these loops together instead of repeating them; consider doing equality on the output arrays instead of collecting the values first
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
+ (
+ "LargeBinary->BinaryView",
+ large_binary,
+ DataType::BinaryView,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_direct),
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_unpack),
+ "{label} (unpack path)"
+ );
+ }
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_both_paths() {
+ // Invalid UTF-8 must behave identically whichever implementation runs, for both
+ // Binary and LargeBinary sources.
+ let mut b32 = BinaryBuilder::new();
+ let mut b64 = GenericBinaryBuilder::<i64>::new();
+ for v in [b"aa".as_slice(), b"bb", &[0xFF, 0xFE], b"dd", b"ee", b"ff"] {
+ b32.append_value(v);
+ b64.append_value(v);
+ }
+ let binary: ArrayRef = Arc::new(b32.finish());
+ let large_binary: ArrayRef = Arc::new(b64.finish());
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let safe = CastOptions {
+ safe: true,
+ ..Default::default()
+ };
+
+ for values in [binary, large_binary] {
+ for keys in [keys_taking_direct_path(), keys_taking_unpack_path()] {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values.clone()).unwrap();
+
+ let err = cast_with_options(&dict, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+
+ let casted = cast_with_options(&dict, &DataType::Utf8View, &safe).unwrap();
+ let got: Vec<_> = casted.as_string_view().iter().collect();
+ // only rows whose key points at the invalid value are nullified
+ assert!(got.iter().all(|v| *v != Some("\u{FFFD}")));
+ assert_eq!(got[0], Some("aa"));
+ }
+ }
+ }
+
+ #[test]
+ fn test_dict_large_utf8_to_utf8view() {
+ // Dict<Int8, LargeUtf8> -> Utf8View, exercising the offset-fit check
+ let values = LargeStringArray::from(vec![
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ Some("hello"),
+ ]);
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0), Some(1)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(dict_array.data_type(), &DataType::Utf8View));
+ let casted = cast(&dict_array, &DataType::Utf8View).unwrap();
+ assert_eq!(casted.data_type(), &DataType::Utf8View);
+
+ let expected = StringViewArray::from(vec![
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ None,
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_large_binary_to_binary_view() {
+ // Dict<Int8, LargeBinary> -> BinaryView, exercising the offset-fit check
+ let mut builder = GenericBinaryBuilder::<i64>::new();
+ builder.append_value(b"hello");
+ builder.append_value(b"world");
+ let values = builder.finish();
+
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(
+ dict_array.data_type(),
+ &DataType::BinaryView
+ ));
+ let casted = cast(&dict_array, &DataType::BinaryView).unwrap();
+ assert_eq!(casted.data_type(), &DataType::BinaryView);
+
+ let expected = BinaryViewArray::from_iter(vec![
+ Some(b"hello".as_slice()),
+ Some(b"world".as_slice()),
+ None,
+ Some(b"hello".as_slice()),
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_utf8_to_binary_view() {
+ // Dict<Int8, Utf8> -> BinaryView cross cast: UTF-8 strings are always valid binary
+ let data = [
+ Some("hello"),
+ Some("repeated"),
+ None,
+ Some("large payload over 12 bytes"),
+ Some("repeated"),
+ ];
+ let values = StringArray::from(data.to_vec());
+ let keys = Int8Array::from_iter([Some(1), Some(0), None, Some(3), None, Some(1), Some(4)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(
+ dict_array.data_type(),
+ &DataType::BinaryView
+ ));
+ let casted = cast(&dict_array, &DataType::BinaryView).unwrap();
+ assert_eq!(casted.data_type(), &DataType::BinaryView);
+
+ let expected = BinaryViewArray::from_iter(vec![
+ data[1], data[0], None, data[3], None, data[1], data[4],
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_valid() {
+ // Dict<Int8, Binary> -> Utf8View cross cast: all values are valid UTF-8
+ let values = BinaryArray::from_iter_values([b"hello".as_slice(), b"world", b"foo"]);
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0), Some(2)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(dict_array.data_type(), &DataType::Utf8View));
+ let casted = cast(&dict_array, &DataType::Utf8View).unwrap();
+ assert_eq!(casted.data_type(), &DataType::Utf8View);
+
+ let result: Vec<_> = casted.as_string_view().iter().collect();
+ assert_eq!(
+ result,
+ vec![
+ Some("hello"),
+ Some("world"),
+ None,
+ Some("hello"),
+ Some("foo")
+ ]
+ );
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_strict() {
+ // Dict<Int8, Binary> -> Utf8View with invalid UTF-8: safe=false returns an error
+ let mut builder = BinaryBuilder::new();
+ builder.append_value(b"valid");
+ builder.append_value([0xFF]); // invalid UTF-8
+ builder.append_value(b"also valid");
+ let values = builder.finish();
+
+ let keys = Int8Array::from_iter([Some(0), Some(1), Some(2)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let err = cast_with_options(&dict_array, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_safe() {
Review Comment:
how is this different from `test_dict_binary_to_utf8view_invalid_utf8_both_paths`
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
+ (
+ "LargeBinary->BinaryView",
+ large_binary,
+ DataType::BinaryView,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_direct),
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_unpack),
+ "{label} (unpack path)"
+ );
+ }
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_both_paths() {
+ // Invalid UTF-8 must behave identically whichever implementation runs, for both
+ // Binary and LargeBinary sources.
+ let mut b32 = BinaryBuilder::new();
+ let mut b64 = GenericBinaryBuilder::<i64>::new();
+ for v in [b"aa".as_slice(), b"bb", &[0xFF, 0xFE], b"dd", b"ee", b"ff"] {
+ b32.append_value(v);
+ b64.append_value(v);
+ }
+ let binary: ArrayRef = Arc::new(b32.finish());
+ let large_binary: ArrayRef = Arc::new(b64.finish());
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let safe = CastOptions {
+ safe: true,
+ ..Default::default()
+ };
+
+ for values in [binary, large_binary] {
+ for keys in [keys_taking_direct_path(), keys_taking_unpack_path()] {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values.clone()).unwrap();
+
+ let err = cast_with_options(&dict, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+
+ let casted = cast_with_options(&dict, &DataType::Utf8View, &safe).unwrap();
+ let got: Vec<_> = casted.as_string_view().iter().collect();
+ // only rows whose key points at the invalid value are nullified
+ assert!(got.iter().all(|v| *v != Some("\u{FFFD}")));
+ assert_eq!(got[0], Some("aa"));
+ }
+ }
+ }
+
+ #[test]
+ fn test_dict_large_utf8_to_utf8view() {
Review Comment:
is this not tested as part of `test_dict_to_view_both_paths_agree()` above? same for the tests below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5109963595
I'll try take a look at this soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3789937763
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -126,6 +229,73 @@ fn dictionary_to_dictionary_cast<K: ArrowDictionaryKeyType>(
Ok(new_array)
}
+/// Cast `Dict<K, Binary>` or `Dict<K, LargeBinary>` to `Utf8View`, validating UTF-8 for each
+/// dictionary value.
+///
+/// Fast path when all values are valid UTF-8: reuses the values buffer without copying.
+/// When some values are invalid and `cast_options.safe` is true, rows pointing to those
+/// values become null. When `cast_options.safe` is false, returns an error immediately.
+fn binary_dict_to_string_view<K: ArrowDictionaryKeyType, O: OffsetSizeTrait>(
+ keys: &PrimitiveArray<K>,
+ values: &GenericByteArray<GenericBinaryType<O>>,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match GenericStringArray::<O>::try_from_binary(values.clone()) {
+ Ok(_) => {
+ // All dictionary values are valid UTF-8: reuse the buffer zero-copy.
+ view_from_dict_values::<K, GenericBinaryType<O>, StringViewType>(keys, values)
+ }
+ Err(e) => {
+ if !cast_options.safe {
+ return Err(e);
Review Comment:
I had a look, **this isn't new behaviour from the pr**. `binary_dict_to_string_view` calls `try_from_binary` on the values, same as `cast_binary_to_string` does, and that validates the whole buffer in one pass without looking at the null buffer, so `dict -> utf8view` and `dict -> utf8` fail on the same input, and even unused invalid values fail too.
imo, i'd leave it as is here. relaxing only utf8view would make the two behave differently, and doing it properly means changing validation for both, which would cost the performance this pr is after. happy to look at that separately if you think it's worth it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5323840003
thanks @Abhisheklearn12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5279508585
hi @Jefffrey, would appreciate your any new feedbacks on this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3792817362
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
+ // Every arm, exercised through both implementations.
+ let long = "a value over twelve bytes";
+ let expect_direct = vec![Some("aa"), Some("dd")];
+ let expect_unpack = vec![
+ Some("aa"),
+ Some("dd"),
+ None,
+ Some("bb"),
+ Some(long),
+ Some("aa"),
+ ];
+ fn as_bytes<'a>(v: &[Option<&'a str>]) -> Vec<Option<&'a [u8]>> {
+ v.iter().map(|s| s.map(|s| s.as_bytes())).collect()
+ }
+
+ let utf8: ArrayRef = Arc::new(StringArray::from(vec!["aa", "bb", long, "dd", "ee", "ff"]));
+ let large_utf8: ArrayRef = Arc::new(LargeStringArray::from(vec![
+ "aa", "bb", long, "dd", "ee", "ff",
+ ]));
+ let binary: ArrayRef = Arc::new(BinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+ let large_binary: ArrayRef = Arc::new(LargeBinaryArray::from_iter_values([
+ b"aa".as_slice(),
+ b"bb",
+ long.as_bytes(),
+ b"dd",
+ b"ee",
+ b"ff",
+ ]));
+
+ // every source type that can reach Utf8View
+ for (label, values, to_type) in [
+ ("Utf8->Utf8View", utf8.clone(), DataType::Utf8View),
+ (
+ "LargeUtf8->Utf8View",
+ large_utf8.clone(),
+ DataType::Utf8View,
+ ),
+ ("Binary->Utf8View", binary.clone(), DataType::Utf8View),
+ (
+ "LargeBinary->Utf8View",
+ large_binary.clone(),
+ DataType::Utf8View,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_string_view().iter().collect::<Vec<_>>(),
+ expect_direct,
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_string_view().iter().collect::<Vec<_>>(),
+ expect_unpack,
+ "{label} (unpack path)"
+ );
+ }
+
+ // every source type that can reach BinaryView
+ for (label, values, to_type) in [
+ ("Utf8->BinaryView", utf8, DataType::BinaryView),
+ ("LargeUtf8->BinaryView", large_utf8, DataType::BinaryView),
+ ("Binary->BinaryView", binary, DataType::BinaryView),
+ (
+ "LargeBinary->BinaryView",
+ large_binary,
+ DataType::BinaryView,
+ ),
+ ] {
+ let direct = cast_dict(values.clone(), keys_taking_direct_path(), &to_type);
+ assert_eq!(
+ direct.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_direct),
+ "{label} (direct path)"
+ );
+ let unpacked = cast_dict(values, keys_taking_unpack_path(), &to_type);
+ assert_eq!(
+ unpacked.as_binary_view().iter().collect::<Vec<_>>(),
+ as_bytes(&expect_unpack),
+ "{label} (unpack path)"
+ );
+ }
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_both_paths() {
+ // Invalid UTF-8 must behave identically whichever implementation runs, for both
+ // Binary and LargeBinary sources.
+ let mut b32 = BinaryBuilder::new();
+ let mut b64 = GenericBinaryBuilder::<i64>::new();
+ for v in [b"aa".as_slice(), b"bb", &[0xFF, 0xFE], b"dd", b"ee", b"ff"] {
+ b32.append_value(v);
+ b64.append_value(v);
+ }
+ let binary: ArrayRef = Arc::new(b32.finish());
+ let large_binary: ArrayRef = Arc::new(b64.finish());
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let safe = CastOptions {
+ safe: true,
+ ..Default::default()
+ };
+
+ for values in [binary, large_binary] {
+ for keys in [keys_taking_direct_path(), keys_taking_unpack_path()] {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values.clone()).unwrap();
+
+ let err = cast_with_options(&dict, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+
+ let casted = cast_with_options(&dict, &DataType::Utf8View, &safe).unwrap();
+ let got: Vec<_> = casted.as_string_view().iter().collect();
+ // only rows whose key points at the invalid value are nullified
+ assert!(got.iter().all(|v| *v != Some("\u{FFFD}")));
+ assert_eq!(got[0], Some("aa"));
+ }
+ }
+ }
+
+ #[test]
+ fn test_dict_large_utf8_to_utf8view() {
+ // Dict<Int8, LargeUtf8> -> Utf8View, exercising the offset-fit check
+ let values = LargeStringArray::from(vec![
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ Some("hello"),
+ ]);
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0), Some(1)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(dict_array.data_type(), &DataType::Utf8View));
+ let casted = cast(&dict_array, &DataType::Utf8View).unwrap();
+ assert_eq!(casted.data_type(), &DataType::Utf8View);
+
+ let expected = StringViewArray::from(vec![
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ None,
+ Some("hello"),
+ Some("large payload over 12 bytes"),
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_large_binary_to_binary_view() {
+ // Dict<Int8, LargeBinary> -> BinaryView, exercising the offset-fit check
+ let mut builder = GenericBinaryBuilder::<i64>::new();
+ builder.append_value(b"hello");
+ builder.append_value(b"world");
+ let values = builder.finish();
+
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(
+ dict_array.data_type(),
+ &DataType::BinaryView
+ ));
+ let casted = cast(&dict_array, &DataType::BinaryView).unwrap();
+ assert_eq!(casted.data_type(), &DataType::BinaryView);
+
+ let expected = BinaryViewArray::from_iter(vec![
+ Some(b"hello".as_slice()),
+ Some(b"world".as_slice()),
+ None,
+ Some(b"hello".as_slice()),
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_utf8_to_binary_view() {
+ // Dict<Int8, Utf8> -> BinaryView cross cast: UTF-8 strings are always valid binary
+ let data = [
+ Some("hello"),
+ Some("repeated"),
+ None,
+ Some("large payload over 12 bytes"),
+ Some("repeated"),
+ ];
+ let values = StringArray::from(data.to_vec());
+ let keys = Int8Array::from_iter([Some(1), Some(0), None, Some(3), None, Some(1), Some(4)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(
+ dict_array.data_type(),
+ &DataType::BinaryView
+ ));
+ let casted = cast(&dict_array, &DataType::BinaryView).unwrap();
+ assert_eq!(casted.data_type(), &DataType::BinaryView);
+
+ let expected = BinaryViewArray::from_iter(vec![
+ data[1], data[0], None, data[3], None, data[1], data[4],
+ ]);
+ assert_eq!(casted.as_ref(), &expected);
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_valid() {
+ // Dict<Int8, Binary> -> Utf8View cross cast: all values are valid UTF-8
+ let values = BinaryArray::from_iter_values([b"hello".as_slice(), b"world", b"foo"]);
+ let keys = Int8Array::from_iter([Some(0), Some(1), None, Some(0), Some(2)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ assert!(can_cast_types(dict_array.data_type(), &DataType::Utf8View));
+ let casted = cast(&dict_array, &DataType::Utf8View).unwrap();
+ assert_eq!(casted.data_type(), &DataType::Utf8View);
+
+ let result: Vec<_> = casted.as_string_view().iter().collect();
+ assert_eq!(
+ result,
+ vec![
+ Some("hello"),
+ Some("world"),
+ None,
+ Some("hello"),
+ Some("foo")
+ ]
+ );
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_strict() {
+ // Dict<Int8, Binary> -> Utf8View with invalid UTF-8: safe=false returns an error
+ let mut builder = BinaryBuilder::new();
+ builder.append_value(b"valid");
+ builder.append_value([0xFF]); // invalid UTF-8
+ builder.append_value(b"also valid");
+ let values = builder.finish();
+
+ let keys = Int8Array::from_iter([Some(0), Some(1), Some(2)]);
+ let dict_array = DictionaryArray::<Int8Type>::try_new(keys, Arc::new(values)).unwrap();
+
+ let strict = CastOptions {
+ safe: false,
+ ..Default::default()
+ };
+ let err = cast_with_options(&dict_array, &DataType::Utf8View, &strict).unwrap_err();
+ assert!(
+ matches!(err, ArrowError::InvalidArgumentError(_)),
+ "expected InvalidArgumentError, got {err:?}"
+ );
+ }
+
+ #[test]
+ fn test_dict_binary_to_utf8view_invalid_utf8_safe() {
Review Comment:
it wasn't, merged into the one test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 closed pull request #10436: perf(arrow-cast): gate Dictionary -> View fast path on cardinality
URL: https://github.com/apache/arrow-rs/pull/10436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5190504589
what benchmarks am i looking at here? cast kernels seems to only have a single related benchmark, for `dict<utf8> -> utf8view`
can we add these benchmarks in a separate PR so we can verify with the bot (ideally keeping the minimum required, e.g. don't need both largestring -> utf8view and largebinary -> binaryview since theyre essentially the same code paths)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5192824126
large variant benchmark probably not needed since theres only one extra check thats not part of the hot loop, so to keep things simple we just need the `dict<utf8> -> utf8view` and `dict<binary> -> utf8view`, preferably with dense & sparse versions for the `dict<utf8> -> utf8view` case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234405658
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234396693)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5234396693-1509-fh5wr 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix/dict-view-cardinality-gate (93a5cb685f0841afd263359c296a1ff6069eb9e3) to bb82f3e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/bb82f3e4fcb99cea6c730e906c067579802593fe..93a5cb685f0841afd263359c296a1ff6069eb9e3)
<details><summary>Run configuration</summary>
```yaml
run benchmark cast_kernels
env:
BENCH_FILTER: "dict.*to.*view"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench cast_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234382638
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234371550)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5234371550-1508-4ggcv 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix/dict-view-cardinality-gate (93a5cb685f0841afd263359c296a1ff6069eb9e3) to bb82f3e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/bb82f3e4fcb99cea6c730e906c067579802593fe..93a5cb685f0841afd263359c296a1ff6069eb9e3)
<details><summary>Run configuration</summary>
```yaml
run benchmark cast_kernels
env:
BENCH_FILTER: "dict.*to.*view"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench cast_kernels
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5190993231
> what benchmarks am i looking at here? cast kernels seems to only have a single related benchmark, for `dict<utf8> -> utf8view`
>
> can we add these benchmarks in a separate PR so we can verify with the bot (ideally keeping the minimum required, e.g. don't need both largestring -> utf8view and largebinary -> binaryview since theyre essentially the same code paths)
well, those numbers came from a local harness rather than anything checked in, which is why they aren't in the repo. i'll add the benchmarks in a separate pr so the bot can verify them.
agreed on keeping the set minimal. `view_from_dict_values` is generic, so `largeutf8 -> utf8view` and `largebinary ->
binaryview` are really the same implementation with different type params. i think the minimum set would be:
- `dict<utf8> -> utf8view`
- `dict<largeutf8> -> utf8view` to cover the i64 offset path
- `dict<binary> -> utf8view`, since that goes through utf8 validation and has a different cost profile
the existing `cast dict to string view` benchmark already covers the dense path. does that sound right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3789965034
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -7639,6 +7639,316 @@ mod tests {
assert_eq!(casted_binary_array.as_ref(), &binary_view_array);
}
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
Review Comment:
also i forgot to say, i left the older dict to view tests (`test_dict_to_view`, `test_string_dicts_to_binary_view`, `test_binary_dicts_to_string_view`) in `mod.rs` since they're already on main, happy to move them in a follow up if you'd rather they all sit together :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5234396693
run benchmark cast_kernels
env:
BENCH_FILTER: dict.*to.*view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3789570998
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -34,23 +34,126 @@ pub(crate) fn dictionary_cast<K: ArrowDictionaryKeyType>(
(_, Dictionary(to_index_type, to_value_type)) => {
dictionary_to_dictionary_cast(array, to_index_type, to_value_type, cast_options)
}
- // `unpack_dictionary` can handle Utf8View/BinaryView types, but incurs unnecessary data
- // copy of the value buffer. Fast path which avoids copying underlying values buffer.
- // TODO: handle LargeUtf8/LargeBinary -> View (need to check offsets can fit)
- // TODO: handle cross types (String -> BinaryView, Binary -> StringView)
- // (need to validate utf8?)
- (Utf8, Utf8View) => view_from_dict_values::<K, Utf8Type, StringViewType>(
- array.keys(),
- array.values().as_string::<i32>(),
- ),
- (Binary, BinaryView) => view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ // There are two ways to produce a view array from a dictionary, and neither of them
+ // copies the values buffer:
+ //
+ // (a) `unpack_dictionary` builds one view per *dictionary value* and then uses `take`,
+ // which gathers 16-byte views and passes the data buffers through untouched.
+ // (b) `view_from_dict_values` builds one view per *row* directly against the values
+ // buffer, skipping the intermediate array entirely.
+ //
+ // (a) is memory-bandwidth-bound -- the gather in `take` benchmarks at or below the cost
+ // of a hand-written `u128` gather -- so it wins once rows reach roughly 0.6x the
+ // dictionary size. (b) reads the payload bytes of every row out of the values buffer to
+ // build each view, which is far more expensive per row, and only pays off when the
+ // dictionary is substantially larger than the array is long: there (a) spends most of
+ // its time building views that no row ever references.
+ //
+ // So take (b) only when it actually wins. Everything else falls through to (a) below.
Review Comment:
```suggestion
// `unpack_dictionary` operates per dictionary value before using take kernel to form
// final view. `view_from_dict_values` builds output view directly per row (key index).
// Based on benchmarking, `view_from_dict_values` is more efficient for sparse dictionaries
// (more dictionary values than there are rows/keys), whilst `unpack_dictionary` is
// more efficient for dense dictionaries, where a sparse dictionary is when rows reach
// roughly 0.6x the dictionary size.
//
// Therefore delegate to the faster method based on the density of the input dictionary.
```
My attempt at cutting down the verbosity a bit, feel free to edit or make suggestions if it doesn't seem as clear or is inaccurate at bits
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -126,6 +229,73 @@ fn dictionary_to_dictionary_cast<K: ArrowDictionaryKeyType>(
Ok(new_array)
}
+/// Cast `Dict<K, Binary>` or `Dict<K, LargeBinary>` to `Utf8View`, validating UTF-8 for each
+/// dictionary value.
+///
+/// Fast path when all values are valid UTF-8: reuses the values buffer without copying.
+/// When some values are invalid and `cast_options.safe` is true, rows pointing to those
+/// values become null. When `cast_options.safe` is false, returns an error immediately.
+fn binary_dict_to_string_view<K: ArrowDictionaryKeyType, O: OffsetSizeTrait>(
+ keys: &PrimitiveArray<K>,
+ values: &GenericByteArray<GenericBinaryType<O>>,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match GenericStringArray::<O>::try_from_binary(values.clone()) {
+ Ok(_) => {
+ // All dictionary values are valid UTF-8: reuse the buffer zero-copy.
+ view_from_dict_values::<K, GenericBinaryType<O>, StringViewType>(keys, values)
+ }
+ Err(e) => {
+ if !cast_options.safe {
+ return Err(e);
Review Comment:
do we need to consider if potentially invalid utf8 is actually masked out by the null buffer of the byte array? this could be overly restrictive for such niche inputs
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -34,23 +34,126 @@ pub(crate) fn dictionary_cast<K: ArrowDictionaryKeyType>(
(_, Dictionary(to_index_type, to_value_type)) => {
dictionary_to_dictionary_cast(array, to_index_type, to_value_type, cast_options)
}
- // `unpack_dictionary` can handle Utf8View/BinaryView types, but incurs unnecessary data
- // copy of the value buffer. Fast path which avoids copying underlying values buffer.
- // TODO: handle LargeUtf8/LargeBinary -> View (need to check offsets can fit)
- // TODO: handle cross types (String -> BinaryView, Binary -> StringView)
- // (need to validate utf8?)
- (Utf8, Utf8View) => view_from_dict_values::<K, Utf8Type, StringViewType>(
- array.keys(),
- array.values().as_string::<i32>(),
- ),
- (Binary, BinaryView) => view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ // There are two ways to produce a view array from a dictionary, and neither of them
+ // copies the values buffer:
+ //
+ // (a) `unpack_dictionary` builds one view per *dictionary value* and then uses `take`,
+ // which gathers 16-byte views and passes the data buffers through untouched.
+ // (b) `view_from_dict_values` builds one view per *row* directly against the values
+ // buffer, skipping the intermediate array entirely.
+ //
+ // (a) is memory-bandwidth-bound -- the gather in `take` benchmarks at or below the cost
+ // of a hand-written `u128` gather -- so it wins once rows reach roughly 0.6x the
+ // dictionary size. (b) reads the payload bytes of every row out of the values buffer to
+ // build each view, which is far more expensive per row, and only pays off when the
+ // dictionary is substantially larger than the array is long: there (a) spends most of
+ // its time building views that no row ever references.
+ //
+ // So take (b) only when it actually wins. Everything else falls through to (a) below.
+ (Utf8, Utf8View) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (Binary, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i32>(),
+ )
+ }
+ // LargeUtf8/LargeBinary additionally require a values buffer small enough to be addressed
+ // by the u32 offset of a view.
+ (LargeUtf8, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ (LargeBinary, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeBinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ )
+ }
+ // Cross casts to a binary view need no validation: valid UTF-8 is valid binary.
+ (Utf8, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (LargeUtf8, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ // Cross casts to a string view require UTF-8 validation of the dictionary values.
+ (Binary, Utf8View) if prefer_direct_views(array) => binary_dict_to_string_view::<K, i32>(
array.keys(),
array.values().as_binary::<i32>(),
+ cast_options,
),
+ (LargeBinary, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ binary_dict_to_string_view::<K, i64>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ cast_options,
+ )
+ }
_ => unpack_dictionary(array, to_type, cast_options),
}
}
+/// Whether building views per row beats `unpack_dictionary` for this array.
+///
+/// `unpack_dictionary` costs `O(values)` to build the intermediate view array plus `O(keys)` for
+/// a bandwidth-bound gather; building views directly costs `O(keys)` but with a much larger
+/// constant, since every row has to read its payload out of the values buffer. The direct path
+/// therefore only wins when the dictionary is substantially larger than the array is long.
+///
+/// The measured crossover sits near `keys ~= 0.6 * values`, so this deliberately switches short
+/// of it rather than at the crossover itself: at `keys * 2 == values` the direct path is still
+/// ahead by 15-30%, which leaves margin for the exact crossover moving with cache size or
+/// microarchitecture. Being wrong in this direction merely forgoes a win; being wrong the other
+/// way is a large regression on the common case.
+#[inline]
+fn prefer_direct_views<K: ArrowDictionaryKeyType>(array: &DictionaryArray<K>) -> bool {
Review Comment:
```suggestion
#[inline]
fn is_sparse<K: ArrowDictionaryKeyType>(array: &DictionaryArray<K>) -> bool {
```
I feel the doc comment is too verbose and repeats what is said above; maybe we can move this method as an inner method to `dictionary_cast` to try colocate the comments to avoid redundancy
also a name like `is_sparse` to me reads better than `prefer_direct_views` because we dont exactly know what `direct_views` means here
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -7639,6 +7639,316 @@ mod tests {
assert_eq!(casted_binary_array.as_ref(), &binary_view_array);
}
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
Review Comment:
it might be better to move this test code into cast/dictionary.rs file, so it lives closer together
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -34,23 +34,126 @@ pub(crate) fn dictionary_cast<K: ArrowDictionaryKeyType>(
(_, Dictionary(to_index_type, to_value_type)) => {
dictionary_to_dictionary_cast(array, to_index_type, to_value_type, cast_options)
}
- // `unpack_dictionary` can handle Utf8View/BinaryView types, but incurs unnecessary data
- // copy of the value buffer. Fast path which avoids copying underlying values buffer.
- // TODO: handle LargeUtf8/LargeBinary -> View (need to check offsets can fit)
- // TODO: handle cross types (String -> BinaryView, Binary -> StringView)
- // (need to validate utf8?)
- (Utf8, Utf8View) => view_from_dict_values::<K, Utf8Type, StringViewType>(
- array.keys(),
- array.values().as_string::<i32>(),
- ),
- (Binary, BinaryView) => view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ // There are two ways to produce a view array from a dictionary, and neither of them
+ // copies the values buffer:
+ //
+ // (a) `unpack_dictionary` builds one view per *dictionary value* and then uses `take`,
+ // which gathers 16-byte views and passes the data buffers through untouched.
+ // (b) `view_from_dict_values` builds one view per *row* directly against the values
+ // buffer, skipping the intermediate array entirely.
+ //
+ // (a) is memory-bandwidth-bound -- the gather in `take` benchmarks at or below the cost
+ // of a hand-written `u128` gather -- so it wins once rows reach roughly 0.6x the
+ // dictionary size. (b) reads the payload bytes of every row out of the values buffer to
+ // build each view, which is far more expensive per row, and only pays off when the
+ // dictionary is substantially larger than the array is long: there (a) spends most of
+ // its time building views that no row ever references.
+ //
+ // So take (b) only when it actually wins. Everything else falls through to (a) below.
+ (Utf8, Utf8View) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (Binary, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i32>(),
+ )
+ }
+ // LargeUtf8/LargeBinary additionally require a values buffer small enough to be addressed
+ // by the u32 offset of a view.
+ (LargeUtf8, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ (LargeBinary, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeBinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ )
+ }
+ // Cross casts to a binary view need no validation: valid UTF-8 is valid binary.
+ (Utf8, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (LargeUtf8, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ // Cross casts to a string view require UTF-8 validation of the dictionary values.
+ (Binary, Utf8View) if prefer_direct_views(array) => binary_dict_to_string_view::<K, i32>(
array.keys(),
array.values().as_binary::<i32>(),
+ cast_options,
),
+ (LargeBinary, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ binary_dict_to_string_view::<K, i64>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ cast_options,
+ )
+ }
_ => unpack_dictionary(array, to_type, cast_options),
}
}
+/// Whether building views per row beats `unpack_dictionary` for this array.
+///
+/// `unpack_dictionary` costs `O(values)` to build the intermediate view array plus `O(keys)` for
+/// a bandwidth-bound gather; building views directly costs `O(keys)` but with a much larger
+/// constant, since every row has to read its payload out of the values buffer. The direct path
+/// therefore only wins when the dictionary is substantially larger than the array is long.
+///
+/// The measured crossover sits near `keys ~= 0.6 * values`, so this deliberately switches short
+/// of it rather than at the crossover itself: at `keys * 2 == values` the direct path is still
+/// ahead by 15-30%, which leaves margin for the exact crossover moving with cache size or
+/// microarchitecture. Being wrong in this direction merely forgoes a win; being wrong the other
+/// way is a large regression on the common case.
+#[inline]
+fn prefer_direct_views<K: ArrowDictionaryKeyType>(array: &DictionaryArray<K>) -> bool {
+ array.keys().len() < array.values().len() / 2
+}
+
+/// Whether the values buffer can back a view array, i.e. it is smaller than 4GiB.
+///
+/// Required because views address their data with a `u32` offset, and
+/// [`GenericByteViewBuilder::append_block`] asserts the block it is handed is smaller than
+/// `u32::MAX`. Failing this check is therefore a panic in the direct path, not an error.
+///
+/// `unpack_dictionary` has no such limit: it reaches
+/// `impl From<&GenericByteArray> for GenericByteViewArray`, which makes the same test and falls
+/// back to copying via `from_iter` when the buffer is too large. So this guards the direct path
+/// only, and the fallback is strictly more capable rather than an equivalent failure.
+///
+/// This deliberately measures the whole buffer rather than the largest live offset: slicing a
+/// byte array slices its offsets but keeps `value_data` intact, so a slice can have small offsets
+/// and still carry an oversized buffer -- and it is the buffer that `append_block` rejects.
+#[inline]
+fn values_buffer_fits_in_view<T: ByteArrayType>(values: &GenericByteArray<T>) -> bool {
+ values.values().len() < u32::MAX as usize
Review Comment:
```suggestion
values.values().len() < i32::MAX as usize
```
- https://github.com/apache/arrow-rs/issues/6172
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -34,23 +34,126 @@ pub(crate) fn dictionary_cast<K: ArrowDictionaryKeyType>(
(_, Dictionary(to_index_type, to_value_type)) => {
dictionary_to_dictionary_cast(array, to_index_type, to_value_type, cast_options)
}
- // `unpack_dictionary` can handle Utf8View/BinaryView types, but incurs unnecessary data
- // copy of the value buffer. Fast path which avoids copying underlying values buffer.
- // TODO: handle LargeUtf8/LargeBinary -> View (need to check offsets can fit)
- // TODO: handle cross types (String -> BinaryView, Binary -> StringView)
- // (need to validate utf8?)
- (Utf8, Utf8View) => view_from_dict_values::<K, Utf8Type, StringViewType>(
- array.keys(),
- array.values().as_string::<i32>(),
- ),
- (Binary, BinaryView) => view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ // There are two ways to produce a view array from a dictionary, and neither of them
+ // copies the values buffer:
+ //
+ // (a) `unpack_dictionary` builds one view per *dictionary value* and then uses `take`,
+ // which gathers 16-byte views and passes the data buffers through untouched.
+ // (b) `view_from_dict_values` builds one view per *row* directly against the values
+ // buffer, skipping the intermediate array entirely.
+ //
+ // (a) is memory-bandwidth-bound -- the gather in `take` benchmarks at or below the cost
+ // of a hand-written `u128` gather -- so it wins once rows reach roughly 0.6x the
+ // dictionary size. (b) reads the payload bytes of every row out of the values buffer to
+ // build each view, which is far more expensive per row, and only pays off when the
+ // dictionary is substantially larger than the array is long: there (a) spends most of
+ // its time building views that no row ever references.
+ //
+ // So take (b) only when it actually wins. Everything else falls through to (a) below.
+ (Utf8, Utf8View) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (Binary, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i32>(),
+ )
+ }
+ // LargeUtf8/LargeBinary additionally require a values buffer small enough to be addressed
+ // by the u32 offset of a view.
Review Comment:
```suggestion
// `view_from_dict_values` directly appends the values buffer as a block using
// `GenericByteViewBuilder::append_block` which asserts length of the buffer; we must
// ensure this assertion holds to use it for large variants which may exceed the max allowable length.
// If we exceed the length we can simply fallback to `unpack_dictionary` which still builds it
// correctly.
```
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -34,23 +34,126 @@ pub(crate) fn dictionary_cast<K: ArrowDictionaryKeyType>(
(_, Dictionary(to_index_type, to_value_type)) => {
dictionary_to_dictionary_cast(array, to_index_type, to_value_type, cast_options)
}
- // `unpack_dictionary` can handle Utf8View/BinaryView types, but incurs unnecessary data
- // copy of the value buffer. Fast path which avoids copying underlying values buffer.
- // TODO: handle LargeUtf8/LargeBinary -> View (need to check offsets can fit)
- // TODO: handle cross types (String -> BinaryView, Binary -> StringView)
- // (need to validate utf8?)
- (Utf8, Utf8View) => view_from_dict_values::<K, Utf8Type, StringViewType>(
- array.keys(),
- array.values().as_string::<i32>(),
- ),
- (Binary, BinaryView) => view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ // There are two ways to produce a view array from a dictionary, and neither of them
+ // copies the values buffer:
+ //
+ // (a) `unpack_dictionary` builds one view per *dictionary value* and then uses `take`,
+ // which gathers 16-byte views and passes the data buffers through untouched.
+ // (b) `view_from_dict_values` builds one view per *row* directly against the values
+ // buffer, skipping the intermediate array entirely.
+ //
+ // (a) is memory-bandwidth-bound -- the gather in `take` benchmarks at or below the cost
+ // of a hand-written `u128` gather -- so it wins once rows reach roughly 0.6x the
+ // dictionary size. (b) reads the payload bytes of every row out of the values buffer to
+ // build each view, which is far more expensive per row, and only pays off when the
+ // dictionary is substantially larger than the array is long: there (a) spends most of
+ // its time building views that no row ever references.
+ //
+ // So take (b) only when it actually wins. Everything else falls through to (a) below.
+ (Utf8, Utf8View) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (Binary, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, BinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i32>(),
+ )
+ }
+ // LargeUtf8/LargeBinary additionally require a values buffer small enough to be addressed
+ // by the u32 offset of a view.
+ (LargeUtf8, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, StringViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ (LargeBinary, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeBinaryType, BinaryViewType>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ )
+ }
+ // Cross casts to a binary view need no validation: valid UTF-8 is valid binary.
+ (Utf8, BinaryView) if prefer_direct_views(array) => {
+ view_from_dict_values::<K, Utf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i32>(),
+ )
+ }
+ (LargeUtf8, BinaryView)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_string::<i64>()) =>
+ {
+ view_from_dict_values::<K, LargeUtf8Type, BinaryViewType>(
+ array.keys(),
+ array.values().as_string::<i64>(),
+ )
+ }
+ // Cross casts to a string view require UTF-8 validation of the dictionary values.
+ (Binary, Utf8View) if prefer_direct_views(array) => binary_dict_to_string_view::<K, i32>(
array.keys(),
array.values().as_binary::<i32>(),
+ cast_options,
),
+ (LargeBinary, Utf8View)
+ if prefer_direct_views(array)
+ && values_buffer_fits_in_view(array.values().as_binary::<i64>()) =>
+ {
+ binary_dict_to_string_view::<K, i64>(
+ array.keys(),
+ array.values().as_binary::<i64>(),
+ cast_options,
+ )
+ }
_ => unpack_dictionary(array, to_type, cast_options),
}
}
+/// Whether building views per row beats `unpack_dictionary` for this array.
+///
+/// `unpack_dictionary` costs `O(values)` to build the intermediate view array plus `O(keys)` for
+/// a bandwidth-bound gather; building views directly costs `O(keys)` but with a much larger
+/// constant, since every row has to read its payload out of the values buffer. The direct path
+/// therefore only wins when the dictionary is substantially larger than the array is long.
+///
+/// The measured crossover sits near `keys ~= 0.6 * values`, so this deliberately switches short
+/// of it rather than at the crossover itself: at `keys * 2 == values` the direct path is still
+/// ahead by 15-30%, which leaves margin for the exact crossover moving with cache size or
+/// microarchitecture. Being wrong in this direction merely forgoes a win; being wrong the other
+/// way is a large regression on the common case.
+#[inline]
+fn prefer_direct_views<K: ArrowDictionaryKeyType>(array: &DictionaryArray<K>) -> bool {
+ array.keys().len() < array.values().len() / 2
+}
+
+/// Whether the values buffer can back a view array, i.e. it is smaller than 4GiB.
+///
+/// Required because views address their data with a `u32` offset, and
+/// [`GenericByteViewBuilder::append_block`] asserts the block it is handed is smaller than
+/// `u32::MAX`. Failing this check is therefore a panic in the direct path, not an error.
+///
+/// `unpack_dictionary` has no such limit: it reaches
+/// `impl From<&GenericByteArray> for GenericByteViewArray`, which makes the same test and falls
+/// back to copying via `from_iter` when the buffer is too large. So this guards the direct path
+/// only, and the fallback is strictly more capable rather than an equivalent failure.
+///
+/// This deliberately measures the whole buffer rather than the largest live offset: slicing a
+/// byte array slices its offsets but keeps `value_data` intact, so a slice can have small offsets
+/// and still carry an oversized buffer -- and it is the buffer that `append_block` rejects.
+#[inline]
Review Comment:
```suggestion
```
same here, dont need to repeat here, easier to just keep it within the actual match arms above per my previous comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3793424558
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -126,6 +229,73 @@ fn dictionary_to_dictionary_cast<K: ArrowDictionaryKeyType>(
Ok(new_array)
}
+/// Cast `Dict<K, Binary>` or `Dict<K, LargeBinary>` to `Utf8View`, validating UTF-8 for each
+/// dictionary value.
+///
+/// Fast path when all values are valid UTF-8: reuses the values buffer without copying.
+/// When some values are invalid and `cast_options.safe` is true, rows pointing to those
+/// values become null. When `cast_options.safe` is false, returns an error immediately.
+fn binary_dict_to_string_view<K: ArrowDictionaryKeyType, O: OffsetSizeTrait>(
+ keys: &PrimitiveArray<K>,
+ values: &GenericByteArray<GenericBinaryType<O>>,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match GenericStringArray::<O>::try_from_binary(values.clone()) {
+ Ok(_) => {
+ // All dictionary values are valid UTF-8: reuse the buffer zero-copy.
+ view_from_dict_values::<K, GenericBinaryType<O>, StringViewType>(keys, values)
+ }
+ Err(e) => {
+ if !cast_options.safe {
+ return Err(e);
Review Comment:
thanks thats a good point, ill raise it as a separate issue
- https://github.com/apache/arrow-rs/issues/10711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5087986467
hi @Jefffrey, I’d love to get your feedback whenever you have time, appreciate it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5153002380
also i re-ran after the split, since removing the checks touched `view_from_dict_values` on the hot path. three runs, max spread 0.9 points.
**existing fast paths on main**
| cast | rows | dict values | main | this PR | change |
|---|---|---|---|---|---|
| `Utf8->Utf8View` | 8,192 | 100 | 25.8 us | 3.4 us | **-86.9%** |
| `Utf8->Utf8View` | 1,000,000 | 1,000 | 3207.1 us | 543.0 us | **-83.1%** |
| `Binary->BinaryView` | 8,192 | 100 | 23.7 us | 3.4 us | **-85.8%** |
| `Binary->BinaryView` | 1,000,000 | 1,000 | 3206.9 us | 544.1 us | **-83.0%** |
**new arms, dictionary larger than the array**
| cast | rows | dict values | main | this PR | change |
|---|---|---|---|---|---|
| `LargeUtf8->Utf8View` | 1,000 | 100,000 | 271.7 us | 3.8 us | **-98.6%** |
| `LargeUtf8->BinaryView` | 10,000 | 1,000,000 | 15428.6 us | 78.2 us | **-99.5%** |
| `LargeBinary->BinaryView` | 1,000 | 100,000 | 273.1 us | 3.9 us | **-98.6%** |
| `Utf8->BinaryView` | 10,000 | 1,000,000 | 2867.6 us | 73.8 us | **-97.4%** |
| `Binary->Utf8View` | 1,000 | 100,000 | 350.1 us | 87.3 us | **-75.1%** |
| `LargeBinary->Utf8View` | 10,000 | 1,000,000 | 4487.2 us | 1652.2 us | **-63.2%** |
`Binary`/`LargeBinary -> Utf8View` gain less because UTF-8 validation of the dictionary values dominates.
dense shapes on the new arms: -0.4% to +2.5%, the gate routes them to `unpack_dictionary`.
both impls compiled into one binary and timed in alternating rounds, outputs asserted equal per cell before timing. i7-11700F.
one caveat: my harness's `main` replica calls the direct path straight for the `Utf8->Utf8View` and `Binary->BinaryView` arms, skipping the `cast_with_options` dispatch the real call pays. that is a constant ~0.5us, so those two rows slightly understate the gain. negligible here, but it would dominate on cells of only a few microseconds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5152892231
> > Happy to split either into its own PR if you'd prefer this one stay purely about the fast path.
>
> could we pull out the bug fix changes into a separate PR from the performance improvements
done @Jefffrey , split out. the null handling and bounds check (bug fixes) are in #10510 now, this one is perf only.
also imo, probably worth merging #10510 first, since this PR extends the direct path to more type combos and would spread the null bug until the fix lands, wdyt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3792820867
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
Review Comment:
yeah, you're right. the name was off since each path had its own hardcoded expectation.
both are now checked against `take(values, keys)` then cast, so they'd fail if they disagree. renamed it to `test_dict_to_view_matches_take_then_cast`, the keys can't be the same since their ratio to values determines the path, so i only varied the key count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#discussion_r3793429190
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -561,3 +701,319 @@ where
}
Ok(Arc::new(b.finish()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ /// Casting a dictionary to a view type has two implementations: building one view per row
+ /// directly against the values buffer, and `unpack_dictionary`. Which one runs depends on
+ /// how the row count compares to the dictionary size, so these helpers pin both branches of
+ /// that choice for each arm.
+ ///
+ /// `values` must have 6 entries; the returned key sets sit either side of the threshold.
+ fn keys_taking_direct_path() -> Int32Array {
+ // 2 keys < 6/2 values -> views are built directly per row
+ Int32Array::from_iter([Some(0), Some(3)])
+ }
+
+ fn keys_taking_unpack_path() -> Int32Array {
+ // 6 keys >= 6/2 values -> unpack_dictionary
+ Int32Array::from_iter([Some(0), Some(3), None, Some(1), Some(2), Some(0)])
+ }
+
+ fn cast_dict(values: ArrayRef, keys: Int32Array, to_type: &DataType) -> ArrayRef {
+ let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
+ assert!(can_cast_types(dict.data_type(), to_type));
+ let casted = cast(&dict, to_type).unwrap();
+ assert_eq!(casted.data_type(), to_type);
+ casted
+ }
+
+ #[test]
+ fn test_dict_to_view_both_paths_agree() {
Review Comment:
> the keys can't be the same since their ratio to values determines the path, so i only varied the key count
we probably could keep the same keys and just vary the dictionary values size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(arrow-cast): gate Dictionary -> View fast path on cardinality [arrow-rs]
Posted by "Abhisheklearn12 (via GitHub)" <gi...@apache.org>.
Abhisheklearn12 commented on PR #10436:
URL: https://github.com/apache/arrow-rs/pull/10436#issuecomment-5230399085
Hi @Jefffrey , I pushed the bench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org