You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "DevMattG (via GitHub)" <gi...@apache.org> on 2026/09/10 14:27:58 UTC
[PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
DevMattG opened a new pull request, #11048:
URL: https://github.com/apache/arrow-rs/pull/11048
# Which issue does this PR close?
- Closes #3478.
# Rationale for this change
`FlightDataEncoder` splits record batches based on a cheap, buffer-memory-size heuristic before IPC encoding. That heuristic can be significantly wrong -- it ignores IPC framing/padding overhead, and undercounts batches with dictionaries or highly skewed row sizes -- so pieces produced by the initial split can still end up well over `max_flight_data_size` once actually encoded.
# What changes are included in this PR?
- After the initial heuristic split, each piece is now IPC-encoded and its *actual* encoded size is checked against `max_flight_data_size` in `FlightDataEncoder::encode_piece_adaptive`. If it's still too large, the piece is re-split using the real size ratio and the sub-pieces are recursively re-checked, so oversized pieces get corrected regardless of why the initial estimate was wrong.
- Dictionaries are queued unconditionally as soon as they're encoded, independent of whether the record batch carrying them ends up split further.
- Fixed `split_batch_for_grpc_response`'s batch-count calculation to round up (`div_ceil`) instead of using floor division, which could previously produce one too few batches.
# Are these changes tested?
Yes. Added/extended unit tests in `arrow-flight/src/encode.rs` covering:
- Repeated/deep adaptive re-splitting under a heavily skewed dictionary size distribution.
- Adaptive re-splitting with both `DictionaryHandling::Resend` and the default `DictionaryHandling::Hydrate`.
- A single row whose own encoded size exceeds `max_flight_data_size` (can't be split further, must still be emitted).
# Are there any user-facing changes?
No breaking API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753268442
run benchmark ipc_writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-6332793062
@DevMattG could you rebase this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753281320
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753267044)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5753267044-2536-fcrzr 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing flight/adaptive-grpc-response-splitting (68c48e94b0bde0b44fb4645730ecb215118f66d1) to f9e02ba (merge-base) [diff](https://github.com/apache/arrow-rs/compare/f9e02ba76ad11e1380b559735ac912c602b604fb..68c48e94b0bde0b44fb4645730ecb215118f66d1)
<details><summary>Run configuration</summary>
```yaml
run benchmark ipc_writer
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench ipc_writer
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753302903
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753267044)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing flight/adaptive-grpc-response-splitting (68c48e94b0bde0b44fb4645730ecb215118f66d1) to f9e02ba (merge-base) [diff](https://github.com/apache/arrow-rs/compare/f9e02ba76ad11e1380b559735ac912c602b604fb..68c48e94b0bde0b44fb4645730ecb215118f66d1)
<details><summary>Run configuration</summary>
```yaml
run benchmark ipc_writer
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group flight_adaptive-grpc-response-splitting main
----- --------------------------------------- ----
arrow_ipc_stream_writer/FileWriter/write_10 1.00 89.9±0.52µs ? ?/sec 1.00 89.7±0.51µs ? ?/sec
arrow_ipc_stream_writer/FileWriter/write_10/dict/delta 1.00 153.8±3.72µs ? ?/sec 1.00 154.5±3.36µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10 1.00 22.1±0.18µs ? ?/sec 1.00 22.1±0.29µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/dict 1.03 69.0±0.47µs ? ?/sec 1.00 66.9±4.28µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/dict/delta 1.00 131.5±3.60µs ? ?/sec 1.03 135.1±3.40µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/zstd 1.00 7.3±0.03ms ? ?/sec 1.00 7.2±0.04ms ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10 1.02 87.8±0.48µs ? ?/sec 1.00 86.4±0.84µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict 1.22 87.5±1.17µs ? ?/sec 1.00 71.7±0.40µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict/delta 1.00 147.1±3.13µs ? ?/sec 1.05 154.9±4.36µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/zstd 1.00 7.2±0.03ms ? ?/sec 1.00 7.2±0.04ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 12.5 MiB |
| Avg memory | 10.5 MiB |
| CPU user | 105.6s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 12.5 MiB |
| Avg memory | 10.5 MiB |
| CPU user | 106.1s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753267044
run benchmark ipc_writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#discussion_r4058352124
##########
arrow-flight/src/encode.rs:
##########
@@ -2034,9 +2061,9 @@ mod tests {
let array = StringArray::from_iter_values((0..1024).map(|i| "*".repeat(i)));
let batch = RecordBatch::try_from_iter(vec![("data", Arc::new(array) as _)]).unwrap();
- // overage is much higher than ideal
+ // improved by adaptively re-splitting on the actual encoded size
// https://github.com/apache/arrow-rs/issues/3478
Review Comment:
if this PR closes this issue we can remove this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "DevMattG (via GitHub)" <gi...@apache.org>.
DevMattG commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-6335701424
> @DevMattG could you rebase this PR
Done. Please let me know if there's something else I should do. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753308261
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753268442)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing flight/adaptive-grpc-response-splitting (68c48e94b0bde0b44fb4645730ecb215118f66d1) to f9e02ba (merge-base) [diff](https://github.com/apache/arrow-rs/compare/f9e02ba76ad11e1380b559735ac912c602b604fb..68c48e94b0bde0b44fb4645730ecb215118f66d1)
<details><summary>Run configuration</summary>
```yaml
run benchmark ipc_writer
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group flight_adaptive-grpc-response-splitting main
----- --------------------------------------- ----
arrow_ipc_stream_writer/FileWriter/write_10 1.02 90.1±0.58µs ? ?/sec 1.00 88.4±0.30µs ? ?/sec
arrow_ipc_stream_writer/FileWriter/write_10/dict/delta 1.01 149.9±3.12µs ? ?/sec 1.00 148.1±2.95µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10 1.00 22.3±0.22µs ? ?/sec 1.00 22.2±0.34µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/dict 1.19 67.6±0.45µs ? ?/sec 1.00 56.9±5.35µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/dict/delta 1.00 128.0±3.25µs ? ?/sec 1.03 131.2±2.81µs ? ?/sec
arrow_ipc_stream_writer/StreamEncoder/encode_10/zstd 1.00 7.2±0.03ms ? ?/sec 1.00 7.1±0.03ms ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10 1.03 88.1±0.33µs ? ?/sec 1.00 85.8±0.47µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict 1.02 86.6±0.92µs ? ?/sec 1.00 84.7±0.40µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/dict/delta 1.00 144.0±3.08µs ? ?/sec 1.03 148.8±3.91µs ? ?/sec
arrow_ipc_stream_writer/StreamWriter/write_10/zstd 1.00 7.2±0.04ms ? ?/sec 1.00 7.2±0.04ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 12.6 MiB |
| Avg memory | 10.5 MiB |
| CPU user | 105.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 12.5 MiB |
| Avg memory | 10.4 MiB |
| CPU user | 105.1s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753287366
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11048#issuecomment-5753268442)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5753268442-2537-g7bdf 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing flight/adaptive-grpc-response-splitting (68c48e94b0bde0b44fb4645730ecb215118f66d1) to f9e02ba (merge-base) [diff](https://github.com/apache/arrow-rs/compare/f9e02ba76ad11e1380b559735ac912c602b604fb..68c48e94b0bde0b44fb4645730ecb215118f66d1)
<details><summary>Run configuration</summary>
```yaml
run benchmark ipc_writer
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench ipc_writer
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "DevMattG (via GitHub)" <gi...@apache.org>.
DevMattG commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5661548742
@Rich-T-kid, while in #10383 you focus on changing the heuristic, this PR introduces a second step after it, that directly evaluates the actual encoded size of the flight batch. If it still exceeds `max_flight_data_size` a re-split is performed. I think this could solve the problem of the inflated sizes after IPC encoding you were addressing here https://github.com/apache/arrow-rs/pull/10383#issuecomment-5018890719.
Maybe combining this PR with yours could be the best solution so far? (at least I could think of)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(arrow-flight): adaptively re-split oversized FlightData based on actual encoded size [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #11048:
URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-5664946793
yea ill close #10383 in favor of this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org