You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/08/27 14:18:55 UTC
[I] [EPIC] Reduce binary size [datafusion]
alamb opened a new issue, #24727:
URL: https://github.com/apache/datafusion/issues/24727
## Is your feature request related to a problem or challenge?
DataFusion generates a large amount of machine code when compiled, which has a few negative effects:
1. **WASM size**: compiled WASM is large (10MB+), slowing page load times (See https://github.com/apache/datafusion/issues/16554)
2. **Build time**: more generated code means more work for LLVM and the linker, so compile times (and thus CI and local iteration) suffer. See https://github.com/apache/datafusion/issues/13814
3. **Binary size**: artifact size (aka binary code) directly affects cold start times, and memory footprint in projects like serverless functions and edge deployments
A significant contributor to code size is **unnecessary monomorphization**, a fancy name for generic functions that get instantiated once per type, but the copies are mostly the same. Common patterns include:
1. **Generics that only call trait methods**: functions like `fn foo<P: ExecutionPlan + ?Sized>(plan: &P)` that never use `P` except through the trait. These compile to N identical copies where a single `&dyn` version would do.
2. **Per-primitive-type instantiation where only the byte width matters**: kernels generic over `ArrowPrimitiveType` that only move, compare-for-equality, or hash native values behave identically for all same-width types (e.g. `Int32`, `UInt32`, `Date32`, `Time32` are all 4-byte bit patterns) and can dispatch on width at runtime instead of instantiating 30+ copies.
3. **Large default trait method bodies**: default methods on widely-implemented traits (e.g. `ScalarUDFImpl`, with 100+ implementations in `datafusion-functions` alone) are monomorphized per implementing type, duplicating the default body each time.
An example of pattern 1 is [`check_default_invariants`](https://github.com/apache/datafusion/blob/66a901f68ddf7ce9c114ee6cb27b3801d4c334e3/datafusion/physical-plan/src/execution_plan.rs#L1642-L1656), which is compiled once for each of the 38 `ExecutionPlan` implementations (~44,000 lines of LLVM IR) even though every copy is identical:
```rust
pub fn check_default_invariants<P: ExecutionPlan + ?Sized>(
plan: &P,
check: InvariantLevel,
) -> Result<(), DataFusionError> {
let children_len = plan.children().len();
check_len!(plan, maintains_input_order, children_len);
check_len!(plan, required_input_ordering, children_len);
check_len!(plan, benefits_from_input_partitioning, children_len);
...
}
```
An example of pattern 2 is [`hash_array_primitive`](https://github.com/apache/datafusion/blob/66a901f68ddf7ce9c114ee6cb27b3801d4c334e3/datafusion/common/src/hash_utils.rs#L306-L333), which is instantiated for all 32 primitive types in every crate that calls `create_hashes` (~12,000 lines of LLVM IR per crate). It only reads native values as bit patterns and hashes them, so the copies for `Int32`, `UInt32`, `Date32`, and `Time32` are byte-identical machine code:
```rust
// condensed; the real function also handles nulls and rehashing
fn hash_array_primitive<T>(
array: &PrimitiveArray<T>,
random_state: &impl HashState,
hashes_buffer: &mut [u64],
rehash: bool,
) where
T: ArrowPrimitiveType<Native: HashValue>,
{
for (hash, &value) in hashes_buffer.iter_mut().zip(array.values().iter()) {
*hash = value.hash_one(random_state);
}
...
}
```
## Early hints / inspiration
@geoffreyclaude and I have been working to keep the size of the IN LIST implementations down as part of https://github.com/apache/datafusion/issues/19241 (he came up with using just bytewidth for FixedSizeBinary in https://github.com/apache/datafusion/pull/24102) and here is an example that reduces code size: https://github.com/apache/datafusion/pull/24687 (removed 67% of the module's generated code with no measurable runtime cost)
and I think we have figured out a general recipe to find and remove this in the codebase:
1. Find candidates with `cargo llvm-lines --release -p <crate> --lib` (rank generic functions by total IR lines × instantiation count)
2. Rewrite the candidate to dispatch at runtime (e.g. per batch, not per row), keeping monomorphized inner loops only where the generated code actually differs
3. Verify with `cargo llvm-lines` before/after and the relevant criterion benchmarks (a size win must not cost measurable performance)
## Describe the solution you'd like
A collection of tickets applying the recipe above to the largest offenders.
## How to measure improvements
Every PR / issue in this epic should report before/after numbers using both of the measurements below, taken once at the merge-base with `main` and once on the branch (see example on "Code size" heading of https://github.com/apache/datafusion/pull/24687).
### 1. Generated code: `cargo llvm-lines`
Counts pre-optimization LLVM IR lines per monomorphized function. Deterministic, and attributes code to specific functions, so it shows exactly which instantiations were removed:
```bash
cargo install cargo-llvm-lines
# Run on main and on your branch:
cargo llvm-lines --release -p <crate> --lib > /tmp/llvm-lines-{main,branch}.txt
# Crate-wide totals (lines and copies):
head -3 /tmp/llvm-lines-main.txt
# Total for the module you changed:
awk '/<module_name>/ {gsub(/[(),%]/,"",$1); sum+=$1; n++}
END {print sum, "IR lines across", n, "functions"}' /tmp/llvm-lines-main.txt
```
Report: module IR lines, module instantiation count, and crate totals, before and after.
### 2. Final binary size: release build
Measures what users actually ship.
```bash
# Gold standard: datafusion-cli (slow: fat LTO takes ~30+ minutes)
cargo build --release --bin datafusion-cli
ls -l target/release/datafusion-cli
# Faster per-crate proxy: any bench binary that links the changed crate
cargo bench --no-run -p <crate> --bench <bench_target>
ls -l target/release/deps/<bench_target>-*
```
Build at the merge-base and on the branch, compare byte sizes.
Note that typically the binary delta will be much smaller than the IR delta as LLVM and the linker deduplicate and dead-code-eliminate aggressively.
## Ideas
Here are some ideas (from an AI tool)
<details><summary>Candidate list (click to expand)</summary>
Initial candidates identified by an `llvm-lines` survey of seven crates (rough IR line totals; final binary impact is smaller but proportional — the PoC measured ~3.3 bytes of binary per IR line):
- [ ] `udaf_default_*` display/schema-name helpers in `datafusion/expr/src/udaf.rs`: generic over `<F: AggregateUDFImpl + ?Sized>` but only call trait methods; ~145K IR lines across 29 UDAF impls (pattern 1)
- [ ] `check_default_invariants` / `check_dynamic_expression_invariants` in `datafusion-physical-plan`: ~69K IR lines across 38 `ExecutionPlan` impls (pattern 1)
- [ ] `IN` list primitive filters (`BranchlessFilter<T>`, `PrimitiveHashSetFilter<T>`, `BitmapFilter<T>`): ~130K IR lines across 27 instantiations; equality-only → width dispatch (pattern 2)
- [ ] `hash_array_primitive` in `datafusion-common`: ~12K IR lines × 32 instantiations *per consuming crate*, ~70K workspace-wide (pattern 2)
- [ ] `first_last` primitive accumulators in `datafusion-functions-aggregate`: ~55K IR lines across 25 instantiations (pattern 2)
- [ ] GROUP BY primitive group values (`PrimitiveGroupValueBuilder<T, NULLABLE>`, `GroupValuesPrimitive<T>`): ~75K IR lines (pattern 2; the `const NULLABLE: bool` parameter doubles every instantiation)
- [ ] TopK aggregate `PrimitiveHashTable<T>`: ~22K IR lines across 32 instantiations (pattern 2)
- [ ] `functions-nested` per-primitive kernels (`array_sort`, `array_min`/`array_max`, `array_has`): ~120K IR lines; the `OffsetSizeTrait` dimension doubles many of these (pattern 2)
- [ ] CASE literal lookup table `PrimitiveIndexMap<T>`: ~25K IR lines across 32 instantiations (pattern 2)
- [ ] Sort-merge primitive cursor streams (`SortPreservingMergeStream` specializations): ~27K IR lines (pattern 2)
- [ ] `ScalarUDFImpl` default method bodies (`return_field_from_args`, `schema_name`, `output_ordering`): ~75K IR lines across 119 impls in `datafusion-functions`, repeated in every crate defining UDFs (pattern 3)
- [ ] https://github.com/apache/datafusion/issues/24658
</details>
## Important caveats
- **Avoid making the code more complicated.** The goal is less generated code, not cleverer source code. Most of these changes will probably reduce code.
- **Float semantics**: `f16`/`f32`/`f64` have distinct equality/hash semantics (`NaN`, `±0.0`), so width-based merging applies to integer-semantics types only; floats likely have to keep their own instantiations
- **Every change must be benchmarked.** The rule from the PoC: a size reduction is acceptable only if the relevant criterion benchmarks show no regression beyond noise.
## Describe alternatives you've considered
Do nothing
## Additional context
- https://github.com/apache/datafusion/issues/16554
- https://github.com/apache/datafusion/issues/13815
- https://github.com/apache/datafusion/issues/13814
- https://github.com/apache/datafusion/pull/24102
- https://github.com/apache/datafusion/pull/24687
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5440760594
Filed the upstream arrow-rs ticket for the largest non-DataFusion item found in the survey (the `PrimitiveArray` `Debug` impl, ~85K IR lines per consuming crate): https://github.com/apache/arrow-rs/issues/10889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5441980968
I made one PR that cuts down the binary size (0.25% -- small but real)
- https://github.com/apache/datafusion/pull/24728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "liningpan (via GitHub)" <gi...@apache.org>.
liningpan commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5682759018
For some reason, `sqlparser` is significantly bloating the datafusion-cli binary and my own project. `cargo bloat` suggests that 6.4MiB of the datafusion-cli binary (release build) is attributed to sqlparser (for my own project, it bloats up to 20 MiB).
Top functions from sqlparser look like this. I wonder if these are inlined copies. The `Statement` struct is 3.6k large
```
> cargo bloat --bin datafusion-cli --release --filter sqlparser -n 100 --full-fn
File .text Size Crate Name
0.0% 0.1% 33.8KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_alter
0.0% 0.1% 32.8KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_create
0.0% 0.0% 26.7KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 26.7KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 26.7KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 26.6KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_alter_table_operation
0.0% 0.0% 19.8KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::ddl::CreateTable as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 19.8KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::ddl::CreateTable as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 19.8KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::ddl::CreateTable as core[663bb9e1522ba5ef]::hash::Hash>::hash::<hashbrown[a17ea828e164abc8]::hasher::DefaultHasher>
0.0% 0.0% 19.4KiB sqlparser <sqlparser[f91aac8db406c26f]::dialect::snowflake::SnowflakeDialect as sqlparser[f91aac8db406c26f]::dialect::Dialect>::parse_statement
0.0% 0.0% 18.4KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_table_factor::{closure#0}
0.0% 0.0% 17.5KiB sqlparser sqlparser[f91aac8db406c26f]::dialect::snowflake::parse_create_table
0.0% 0.0% 17.4KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::fmt::Display>::fmt
0.0% 0.0% 17.4KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_create_table
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 17.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::clone::Clone>::clone
0.0% 0.0% 15.3KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_optional_create_table_config
0.0% 0.0% 15.0KiB sqlparser <sqlparser[f91aac8db406c26f]::tokenizer::Tokenizer>::next_token
0.0% 0.0% 14.7KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_prefix::{closure#1}
0.0% 0.0% 13.9KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.9KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.9KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.9KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::Statement as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.6KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_grant_deny_revoke_privileges_objects
0.0% 0.0% 13.2KiB sqlparser <sqlparser[f91aac8db406c26f]::parser::Parser>::parse_select
0.0% 0.0% 13.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::query::TableFactor as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::query::TableFactor as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::query::TableFactor as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
0.0% 0.0% 13.1KiB sqlparser <sqlparser[f91aac8db406c26f]::ast::query::TableFactor as core[663bb9e1522ba5ef]::hash::Hash>::hash::<std[5d97c59e5e5fafcc]::hash::random::DefaultHasher>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5589858731
> Are you suggesting I add it to somewhere else?
I was actually thinking if it makes sense to have a CI check for any PRs to measure numbers before the merge, and set the binary size metric as the recommendation/requirement. Given that most of PRs are AI generated it might bring some benefit on early capture? WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5686669776
> Top functions from sqlparser look like this. I wonder if these are inlined copies. The Statement struct is 3.6k large
Sounds like a good thing to explore. Maybe worth a ticket upstream to sqlparser-rs 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5450708686
I said this before somewhere in this issue tracker, but I think our liberal use of `impl` in function signatures also contributes to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "srv1n (via GitHub) via github" <gi...@datafusion.apache.org>.
srv1n commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-6623271519
A data point on sqlparser size. datafusion_expr::Expr embeds AST nodes (Expr::Wildcard → WildcardOptions → PlannedReplaceSelectItem → ReplaceSelectElement → sqlparser::ast::Expr), so its derives reach into the AST.
To measure that, I built one small program (parse a statement, then clone, compare, hash and debug-print an Expr) with datafusion-expr 54.1 with and without its sql feature, keeping sqlparser 0.62 linked in both. Fat LTO, one codegen unit, macOS arm64.
__text went from 3.50 MB to 4.74 MB, and symbols naming sqlparser from 1.7 MB to 3.0 MB. sqlparser's Hash code went from none to 0.2 MB. So in this build the Expr link costs about 1.2 MB, which is real but modest, and removing Expr::Wildcard (per the TODO in expr.rs) should recover some of it. It doesn't explain a 20 MiB figure, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5575016484
> Every PR / issue in this epic should report before/after numbers using both of the measurements below, taken once at the merge-base with main and once on the branch
Considering AI nature of created PRs its prob makes sense to have a CI rule to check this on each PR to make PRs more effective 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5589247782
> > Every PR / issue in this epic should report before/after numbers using both of the measurements below, taken once at the merge-base with main and once on the branch
>
> Considering AI nature of created PRs it prob makes sense to have a CI rule to check numbers on each PR overall in this repo to make PRs more effective 🤔
I added this to the issue description:
> Every PR / issue in this epic should report before/after numbers using both of the measurements below, taken once at the merge-base with main and once on the branch (see example on "Code size" heading of https://github.com/apache/datafusion/pull/24687).
Are you suggesting I add it to somewhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Reduce binary size [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24727:
URL: https://github.com/apache/datafusion/issues/24727#issuecomment-5591195026
> > Are you suggesting I add it to somewhere else?
>
> I was actually thinking if it makes sense to have a CI check for any PRs to measure numbers before the merge, and set the binary size metric as the recommendation/requirement. Given that most of PRs are AI generated it might bring some benefit on early capture? WDYT?
I think it is a good idea in theory -- however, I worry about the overall amount of additional CI time. For me it takes 15 minutes to build a release build of `datafusion-cli`.
one thing we could probably do pretty easily is make a script that measured the size for each merge to main, and then posted a report on the PR after the fact (rather than part of CI) 🤔 I bet that woud take like 10 minutes with an agent (and then a few days to run the actual builds 🤔 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org