You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by bl...@apache.org on 2025/10/23 19:26:19 UTC
(datafusion-sandbox) branch main updated: Update main (#21)
This is an automated email from the ASF dual-hosted git repository.
blaginin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-sandbox.git
The following commit(s) were added to refs/heads/main by this push:
new d2bbb91d0 Update main (#21)
d2bbb91d0 is described below
commit d2bbb91d03365c35b2c5b9badb83bed933f3b5ab
Author: Dmitrii Blaginin <dm...@blaginin.me>
AuthorDate: Thu Oct 23 20:26:13 2025 +0100
Update main (#21)
* Use `Display` formatting of `DataType`:s in error messages (#17565)
* Use Display formatting for DataTypes where I could find them
* fix
* More places
* Less Debug
* Cargo fmt
* More cleanup
* Plural types as Display
* Fixes
* Update some more tests and error messages
* Update test snapshot
* last (?) fixes
* update another slt
* Update instructions on how to run the tests
* Ignore pending snapshot files in .gitignore
* Running all the tests is so slow
* just a trailing space
* Update another test
* Fix markdown formatting
* Improve Display for NativeType
* Update code related to error reporting of NativeType
* Revert some formatting
* fixelyfix
* Another snapshot update
* docs: Move Google Summer of Code 2025 pages to a section (#17504)
* Move GSOC content to its own section
* Update to 20205
* feat: Add `OR REPLACE` to creating external tables (#17580)
* feat: Add `OR REPLACE` to creating external tables
* regen
* fmt
* make more explicit + add tests
* clipy fix
---------
Co-authored-by: Dmitrii Blaginin <dm...@blaginin.me>
* `avg(distinct)` support for decimal types (#17560)
* chore: mv `DistinctSumAccumulator` to common
* feat: add avg distinct support for float64 type
* chore: fmt
* refactor: update import for DataType in Float64DistinctAvgAccumulator and remove unused sum_distinct module
* feat: add avg distinct support for float64 type
* feat: add avg distinct support for decimal
* feat: more test for avg distinct in rust api
* Remove DataFrame API tests for avg(distinct)
* Remove proto test
* Fix merge errors
* Refactoring
* Minor cleanup
* Decimal slt tests for avg(distinct)
* Fix state_fields for decimal distinct avg
---------
Co-authored-by: YuNing Chen <ad...@ynchen.me>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
Co-authored-by: Dmitrii Blaginin <dm...@blaginin.me>
* chore(deps): bump taiki-e/install-action from 2.61.8 to 2.61.9 (#17640)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.61.8 to 2.61.9.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/2fdc5fd6ac805b0f8256893bd4c807bcb666af00...8ea32481661d5e04d602f215b94f17e4014b44f9)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.61.9
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump Swatinem/rust-cache from 2.8.0 to 2.8.1 (#17641)
Bumps [Swatinem/rust-cache](https://github.com/swatinem/rust-cache) from 2.8.0 to 2.8.1.
- [Release notes](https://github.com/swatinem/rust-cache/releases)
- [Changelog](https://github.com/Swatinem/rust-cache/blob/master/CHANGELOG.md)
- [Commits](https://github.com/swatinem/rust-cache/compare/98c8021b550208e191a6a3145459bfc9fb29c4c0...f13886b937689c021905a6b90929199931d60db1)
---
updated-dependencies:
- dependency-name: Swatinem/rust-cache
dependency-version: 2.8.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Validate the memory consumption in SPM created by multi level merge (#17029)
* use GreedyMemoryPool for sanity check
* validate whether batch read from spill exceeds max_record_batch_mem
* replace err with warn log
* fix(SubqueryAlias): use maybe_project_redundant_column (#17478)
* fix(SubqueryAlias): use maybe_project_redundant_column
Fixes #17405
* chore: format
* ci: retry
* chore(SubqueryAlias): restructore duplicate detection and add tests
* docs: add examples and context to the reproducer
* minor: Ensure `datafusion-sql` package dependencies have `sql` flag (#17644)
* optimizer: Rewrite `IS NOT DISTINCT FROM` joins as Hash Joins (#17319)
* optimizer: Convert to Hash Join for join predicates like 'a IS NOT DISTINCT FROM b'
* drop tables in slt
* fix rust doc
* Update datafusion/optimizer/src/extract_equijoin_predicate.rs
Co-authored-by: Jonathan Chen <ch...@gmail.com>
* Update datafusion/optimizer/src/extract_equijoin_predicate.rs
* Update datafusion/sqllogictest/test_files/join_is_not_distinct_from.slt
Co-authored-by: Nga Tran <ng...@live.com>
* review: more tests and better error message
* review: improve doc
---------
Co-authored-by: Jonathan Chen <ch...@gmail.com>
Co-authored-by: Nga Tran <ng...@live.com>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Upgrade to arrow 56.1.0 (#17275)
* Update to arrow/parquet 56.1.0
* Adjust for new parquet sizes, update for deprecated API
* Thread through max_predicate_cache_size, add test
* fix: Preserves field metadata when creating logical plan for VALUES expression (#17525)
* [ISSUE 17425] Initial attempt to fix this problem
* Add tests for the fix
* Require that the metadata of values in VALUES clause must be identical
* fix merge error
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* chore(deps): bump serde from 1.0.223 to 1.0.225 (#17614)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.223 to 1.0.225.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.223...v1.0.225)
---
updated-dependencies:
- dependency-name: serde
dependency-version: 1.0.225
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
Co-authored-by: Dmitrii Blaginin <dm...@blaginin.me>
* chore: Update dynamic filter formatting (#17647)
* chore: update dynamic filter formatting to indicate expr is placeholder
* update tests
* update tests
* chore(deps): bump taiki-e/install-action from 2.61.9 to 2.61.10 (#17660)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.61.9 to 2.61.10.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/8ea32481661d5e04d602f215b94f17e4014b44f9...0aa4f22591557b744fe31e55dbfcdfea74a073f7)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.61.10
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* proto: don't include parquet feature by default (#17577)
* feat: add support for RightAnti and RightSemi join types (#17604)
Closes #17603
* minor: Ensure `proto` crate has datetime & unicode expr flags in datafusion dev dependency (#17656)
* minor: Ensure `proto` crate has datetime & unicode expr flags in datafusion dev dependency
* toml formatting
* chore(deps): bump indexmap from 2.11.3 to 2.11.4 (#17661)
Bumps [indexmap](https://github.com/indexmap-rs/indexmap) from 2.11.3 to 2.11.4.
- [Changelog](https://github.com/indexmap-rs/indexmap/blob/main/RELEASES.md)
- [Commits](https://github.com/indexmap-rs/indexmap/compare/2.11.3...2.11.4)
---
updated-dependencies:
- dependency-name: indexmap
dependency-version: 2.11.4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* docs: add xorq to list of known users (#17668)
* Introduce `TypeSignatureClass::Binary` to allow accepting arbitrarily sized `FixedSizeBinary` arguments (#17531)
* Introduce wildcard const for FixedSizeBinary type signature
* Add Binary to TypeSignatureClass
* Remove FIXED_SIZE_BINARY_WILDCARD
* docs: deduplicate links in `introduction.md` (#17669)
* docs: deduplicate links in `introduction.md`
* Further simplifications
* Fix
* Add explicit PMC/committers list to governance docs page (#17574)
* Add committers explicitly to governance page, with script
* add license header
* Update Wes McKinney's affiliation in governance.md
* Update adriangb's affiliation
* Update affiliation
* Andy Grove Affiliation
* Update Qi Zhu affiliation
* Updatd linwei's info
* Update docs/source/contributor-guide/governance.md
* Update docs/source/contributor-guide/governance.md
* Apply suggestions from code review
Co-authored-by: Oleks V <co...@users.noreply.github.com>
Co-authored-by: Liang-Chi Hsieh <vi...@gmail.com>
* Apply suggestions from code review
Co-authored-by: Alex Huang <hu...@gmail.com>
Co-authored-by: Yang Jiang <ji...@163.com>
Co-authored-by: Yongting You <20...@gmail.com>
* Apply suggestions from code review
Co-authored-by: Yijie Shen <he...@gmail.com>
* Apply suggestions from code review
* Apply suggestions from code review
Co-authored-by: Brent Gardner <bg...@squarelabs.net>
Co-authored-by: Dmitrii Blaginin <gi...@blaginin.me>
Co-authored-by: Jax Liu <li...@gmail.com>
Co-authored-by: Ifeanyi Ubah <if...@yahoo.com>
* Apply suggestions from code review
Co-authored-by: Will Jones <wi...@gmail.com>
* Clarify what is updated in the script
* Apply suggestions from code review
Co-authored-by: Paddy Horan <57...@users.noreply.github.com>
Co-authored-by: Dan Harris <13...@users.noreply.github.com>
* Update docs/source/contributor-guide/governance.md
* Update docs/source/contributor-guide/governance.md
Co-authored-by: Parth Chandra <pa...@apache.org>
* Update docs/source/contributor-guide/governance.md
* prettier
---------
Co-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Adrian Garcia Badaracco <17...@users.noreply.github.com>
Co-authored-by: Mustafa Akur <ak...@gmail.com>
Co-authored-by: Qi Zhu <82...@qq.com>
Co-authored-by: 张林伟 <le...@gmail.com>
Co-authored-by: xudong.w <wx...@gmail.com>
Co-authored-by: Oleks V <co...@users.noreply.github.com>
Co-authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Co-authored-by: Alex Huang <hu...@gmail.com>
Co-authored-by: Yang Jiang <ji...@163.com>
Co-authored-by: Yongting You <20...@gmail.com>
Co-authored-by: Yijie Shen <he...@gmail.com>
Co-authored-by: Brent Gardner <bg...@squarelabs.net>
Co-authored-by: Dmitrii Blaginin <gi...@blaginin.me>
Co-authored-by: Jax Liu <li...@gmail.com>
Co-authored-by: Ifeanyi Ubah <if...@yahoo.com>
Co-authored-by: Will Jones <wi...@gmail.com>
Co-authored-by: Paddy Horan <57...@users.noreply.github.com>
Co-authored-by: Dan Harris <13...@users.noreply.github.com>
Co-authored-by: Ruihang Xia <wa...@gmail.com>
Co-authored-by: Parth Chandra <pa...@apache.org>
* fix: Ignore governance doc from typos (#17678)
* Support Decimal32/64 types (#17501)
* Support Decimal32/64 types
* Fix bugs, tests, handle more aggregate functions and schema
* Fill out more parts in expr,common and expr-common
* Some stragglers and overlooked corners
* Actually commit the avg_distinct support
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* minor: Improve hygiene for `datafusion-functions` macros (#17638)
* feat(small): Display `NullEquality` in join executor's `EXPLAIN` output (#17664)
* Clarify null-equal explain expectations
* Format null equality display strings
* fix test
* review: more concise message
* review: more concise message
* Custom timestamp format for DuckDB (#17653)
* feat(substrait): add time literal support (#17655)
Adds support for `ScalarValue::Time64Microsecond` and `ScalarValue::Time64Nanosecond` to be converted to and from Substrait literals. This includes the `PrecisionTime` literal type and specific `TIME_64_TYPE_VARIATION_REF` for 6-digit (microseconds) and 9-digit (nanoseconds) precision.
Co-authored-by: Bruno Volpato <br...@datadoghq.com>
* Support LargeList for array_sort (#17657)
* Support FixedSizeList for array_except (#17658)
* fix: null padding for `array_reverse` on `FixedSizeList` (#17673)
* fix: array_reverse with null
* update
* update
* chore: refactor array fn signatures & add more slt tests (#17672)
* Support FixedSizeList for array_to_string (#17666)
* fix: correct statistics for `NestedLoopJoinExec` (#17680)
* fix: correct statistics for nestedloopexec
* chore: update comment
* minor: add SQLancer fuzzed SLT case for natural joins (#17683)
* chore: Upgrade Rust version to 1.90.0 (#17677)
* chore: bump workspace rust version to 1.90.0
* fix clippy errors
* fix clippy errors
* try using dedicate runner temp space
* retrigger
* inspect disk usage
* split build/run
* disable debug info in ci profile
* revert ci changes
* Support FixedSizeList for array_position (#17659)
* chore(deps): bump the proto group with 2 updates (#16806)
* chore(deps): bump the proto group with 2 updates
Bumps the proto group with 2 updates: [pbjson-build](https://github.com/influxdata/pbjson) and [prost-build](https://github.com/tokio-rs/prost).
Updates `pbjson-build` from 0.7.0 to 0.8.0
- [Commits](https://github.com/influxdata/pbjson/commits)
Updates `prost-build` from 0.13.5 to 0.14.1
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Changelog](https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.13.5...v0.14.1)
---
updated-dependencies:
- dependency-name: pbjson-build
dependency-version: 0.8.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: proto
- dependency-name: prost-build
dependency-version: 0.14.1
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: proto
...
Signed-off-by: dependabot[bot] <su...@github.com>
* Regen protos
---------
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
Co-authored-by: Jefffrey <je...@gmail.com>
* feat(spark): implement Spark `make_interval` function (#17424)
* feat(spark): implement Spark make_interval function
* fix name length
* add doc
* add doc and change test, need more test
* fmt
* add test and doc, need to work in overflow
* clippy
* empty params
* test ok IntervalMonthDayNano::new(0, 0, 0) in unit test
* line blank
* fix doc table select
* dont panic
* update test and not panic fmt
* review
* review fix test failure
* review fix test failure format simple string
* test uncomment and link
* return test (empty)
* changes review
* all overflow null
* all overflow null fix fmt
* changes review
* changes review clippy
* refactor move
* fix error doc date_sub
* clean slt
* no space device
* chore: Update READMEs of crates to be more consistent (#17691)
* chore: Update READMEs of crates to be more consistent
* Add some more Apache project links
* Minor formatting
* Formatting
* Update datafusion/pruning/README.md
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* suggestion
* formatting
* formatting
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* chore: update a bunch of dependencies (#17708)
* chore: fix wasm-pack installation link in wasmtest README (#17704)
* Support FixedSizeList for array_slice via coercion to List (#17667)
* docs: Remove disclaimer that `datafusion` 50.0.0 is not released (#17695)
* docs: Remove disclaimer that datafusion 50.0.0 is not released
* Add section about 51.0.0
* chore(deps): bump taiki-e/install-action from 2.61.10 to 2.62.1 (#17710)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.61.10 to 2.62.1.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/0aa4f22591557b744fe31e55dbfcdfea74a073f7...d6912b47771be2c443ec90dbb3d28e023987e782)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.1
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* perf: Improve the performance of WINDOW functions with many partitions (#17528)
* perf: Improve the performance of WINDOW functions with many partitions
* Improve variable name in calculate_n_out_row
* fix: Partial AggregateMode will generate duplicate field names which will fail DFSchema construct (#17706)
* fix: Partial AggregateMode will generate duplicate field names which will fail DFSchema construct
* Update datafusion/common/src/dfschema.rs
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* fmt
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* feat: expose `udafs` and `udwfs` methods on `FunctionRegistry` (#17650)
* expose udafs and udwfs method on `FunctionRegistry`
* fix doc test
* add default implementations not to trigger
backward incompatible change for others
* Support remaining substrait time literal variations (#17707)
* Bump MSRV to 1.87.0 (#17724)
* Bump MSRV to 1.87.0
* automatic code fixes
* Add upgrading entry
* Avoid redundant Schema clones (#17643)
* Collocate variants of From DFSchema to Schema
* Remove duplicated logic for obtaining Schema from DFSchema
* Remove Arc clone in hash_nested_array
* Avoid redundant Schema clones
* Avoid some Field clones
* make arc clones explicit
* retract the new From
* empty: roll the dice 🎲
* Use github link instead of relative link to optimizer_rule.rs in query-optimizer.md (#17723)
* Move misplaced upgrading entry about MSRV (#17727)
* Introduce `avg_distinct()` and `sum_distinct()` functions to DataFrame API (#17536)
* Introduce `avg_distinct()` and `sum_distinct()` functions to DataFrame API
* Add to roundtrip proto tests
* Support `WHERE`, `ORDER BY`, `LIMIT`, `SELECT`, `EXTEND` pipe operators (#17278)
* support WHERE pipe operator
* support order by
* support limit
* select pipe
* extend support
* document supported pipe operators in user guide
* fmt
* fix where pipe before extend
* don't rebind
* remove clone
* move docs into select.md
* avoid confusion by removing `>` in examples
---------
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* doc: add missing examples for multiple math functions (#17018)
* Update Scalar_functions.md
* pretier fix
* Updated files
* Updated Scalar functions
* Update datafusion/functions/src/math/log.rs
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* Update datafusion/functions/src/math/monotonicity.rs
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* Update datafusion/functions/src/math/monotonicity.rs
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* Update datafusion/functions/src/math/nans.rs
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* Update datafusion/functions/src/math/nanvl.rs
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* Fix tanh example to be tanh not trunc
* Run update_function_docs.sh
---------
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* feat: support for null, date, and timestamp types in approx_distinct (#17618)
* feat: let approx_distinct handle null, date and timestamp types
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
* chore: update testing submodule
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
* feat: supports time type and refactor NullHLLAccumulator
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
* bump arrow-testing submodule
---------
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
Co-authored-by: Jefffrey <je...@gmail.com>
* fix(agg/corr): return NULL when variance is zero or samples < 2 (#17621)
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
* chore(deps): bump taiki-e/install-action from 2.62.1 to 2.62.4 (#17739)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.1 to 2.62.4.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/d6912b47771be2c443ec90dbb3d28e023987e782...5597bc27da443ba8bf9a3bc4e5459ea59177de42)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump tempfile from 3.22.0 to 3.23.0 (#17741)
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.22.0 to 3.23.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.22.0...v3.23.0)
---
updated-dependencies:
- dependency-name: tempfile
dependency-version: 3.23.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore: make `LimitPushPastWindows` public (#17736)
* fix: Remove parquet encryption feature from root deps (#17700)
This fix relates to issue #16650 by completing #16649 .
* fix: Remove datafusion-macros's dependency on datafusion-expr (#17688)
* Remove datafusion-macros's dependency on datafusion-expr
* Re-export
* chore: remove homebrew publish instructions from release steps (#17735)
* minor: create `OptimizerContext` with provided `ConfigOptions` (#17742)
* Improve documentation for ordered set aggregate functions (#17744)
* docs: fix sidebar overlapping table on configuration page on website (#17738)
* solved bug
* fix:modified css for table overlapping
* Add support for calling async UDF as aggregation expression (#17620)
* Add support for calling async UDF as aggregation expression
Fixes https://github.com/apache/datafusion/issues/17619
* add explain plans
* chore(deps): bump taiki-e/install-action from 2.62.4 to 2.62.5 (#17750)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.4 to 2.62.5.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/5597bc27da443ba8bf9a3bc4e5459ea59177de42...6f69ec9970ed0c500b1b76d648e05c4c7e0e5671)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.5
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* (fix): Lag function creates unwanted projection (#17630) (#17639)
* fix: Not adding generated windown expr resulting column twice (#17630)
* Making clippy happier
* Support `LargeList` in `array_has` simplification to `InList` (#17732)
* Support `LargeList` in `array_has` simplification to `InList`
* refactoring
* chore(deps): bump wasm-bindgen-test from 0.3.51 to 0.3.53 (#17642)
* chore(deps): bump wasm-bindgen-test from 0.3.51 to 0.3.53
Bumps [wasm-bindgen-test](https://github.com/wasm-bindgen/wasm-bindgen) from 0.3.51 to 0.3.53.
- [Release notes](https://github.com/wasm-bindgen/wasm-bindgen/releases)
- [Changelog](https://github.com/wasm-bindgen/wasm-bindgen/blob/main/CHANGELOG.md)
- [Commits](https://github.com/wasm-bindgen/wasm-bindgen/commits)
---
updated-dependencies:
- dependency-name: wasm-bindgen-test
dependency-version: 0.3.53
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
* testing setting WASM_BINDGEN_TEST_TIMEOUT
* more testing
* more testing
* more testing
* more testing
* more testing
* testing
* testing
* testing
* testing
* whoops
* whoops
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* testing
* problem commit
* please let this work
* oops
* test 0.3.53
* fix
---------
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
Co-authored-by: Jeffrey Vo <je...@gmail.com>
* feat: support `Utf8View` for more args of `regexp_replace` (#17195)
* Stash changes.
* Signature cleanup, more test scenarios.
* Minor test renaming.
* Simplify signature.
* Update tests.
* Signature change for binary input support.
* Return type changes for binary.
* Stash.
* Stash.
* Stash.
* Stash.
* Fix regx bench.
* Clippy.
* Fix bench regx.
* Refactor signature. I need to remove the match arms that aren't used anymore, update the .slt test for string_view.slt, and understand why String(3) and String(4) is not equivalent to this.
* Remove unnecessary match arms.
* Update string_view slt test.
* Reduce diff by returning to single function with a match arm instead of two.
* Simplify template args.
* Fix benchmark compilation.
* Address PR feedback.
* feat(spark): implement Spark `map` function `map_from_arrays` (#17456)
* feat(spark): implement Spark `map` function `map_from_arrays`
* chore: add test with nested `map_from_arrays` calls, refactor map_deduplicate_keys to remove unnesessary variables and array slices
* fix: clippy warning
* fix: null and different size input lists treatment, chore: move common map funcs to utils.rs, add more tests
* fix: typo
* fix: clippy docstring warning
* chore: move more helpers needed for multiple map functions to utils
* chore: add multi-row tests
* fix: null values treatment
* fix: docstring warnings
* chore(deps): bump object_store from 0.12.3 to 0.12.4 (#17753)
Bumps [object_store](https://github.com/apache/arrow-rs-object-store) from 0.12.3 to 0.12.4.
- [Changelog](https://github.com/apache/arrow-rs-object-store/blob/main/CHANGELOG-old.md)
- [Commits](https://github.com/apache/arrow-rs-object-store/compare/v0.12.3...v0.12.4)
---
updated-dependencies:
- dependency-name: object_store
dependency-version: 0.12.4
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Update `arrow` / `parquet` to 56.2.0 (#17631)
* temp update to arrow 56.2.0 pin
* Update to 56.2.0
* Use released arrow
* Update cargo.lock
* fix lock
* chore(deps): bump taiki-e/install-action from 2.62.5 to 2.62.6 (#17766)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.5 to 2.62.6.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/6f69ec9970ed0c500b1b76d648e05c4c7e0e5671...4575ae687efd0e2c78240087f26013fb2484987f)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.6
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Keep aggregate udaf schema names unique when missing an order-by (#17731)
* test: reproducer of bug
* fix: make schema names unique for approx_percentile_cont
* test: regression test is now resolved
* feat : Display function alias in output column name (#17690)
* display function's alias name in output column
* Update function.rs
* updated verbose name format
* simplify alias logic and removing args clone
* Support join cardinality estimation less conservatively (#17476)
* Support join cardinality estimation if distinct_count is set
Currently we require max and min to be set, as they might be used to
estimate the distinct count. This is unnecessarily conservative if
distinct_count has actually been provided, in which case max and min
won't be used at all and the presence of max or min has no influence
over how good of an estimate it is.
* Update datafusion/physical-plan/src/joins/utils.rs
Co-authored-by: Piotr Findeisen <pi...@gmail.com>
* Update tests
* Calculate cardinality even if distinct or min/max not provided
---------
Co-authored-by: Piotr Findeisen <pi...@gmail.com>
* chore(deps): bump libc from 0.2.175 to 0.2.176 (#17767)
Bumps [libc](https://github.com/rust-lang/libc) from 0.2.175 to 0.2.176.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Changelog](https://github.com/rust-lang/libc/blob/0.2.176/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.175...0.2.176)
---
updated-dependencies:
- dependency-name: libc
dependency-version: 0.2.176
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump postgres-types from 0.2.9 to 0.2.10 (#17768)
Bumps [postgres-types](https://github.com/rust-postgres/rust-postgres) from 0.2.9 to 0.2.10.
- [Release notes](https://github.com/rust-postgres/rust-postgres/releases)
- [Commits](https://github.com/rust-postgres/rust-postgres/compare/postgres-types-v0.2.9...postgres-types-v0.2.10)
---
updated-dependencies:
- dependency-name: postgres-types
dependency-version: 0.2.10
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Use `Expr::qualified_name()` and `Column::new()` to extract partition keys from window and aggregate operators (#17757)
* Use `Expr::qualified_name()` and `Column::new()` to extract partition keys
Using `Expr::schema_name()` and `Column::from_qualified_name()` could
incorrectly parse the column name.
* Use `Expr::qualified_name()` to extract group by keys
* Retrain dataframe tests with filters and aggregates
* Prevent exponential planning time for Window functions - v2 (#17684)
* fix
* Update mod.rs
* Update mod.rs
* Update mod.rs
* tests copied from v1 pr
* test case from review comment
https://github.com/apache/datafusion/pull/17684#discussion_r2366146307
* one more test case
* Update mod.rs
* Update datafusion/physical-plan/src/windows/mod.rs
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Update datafusion/physical-plan/src/windows/mod.rs
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Update mod.rs
* Update mod.rs
---------
Co-authored-by: Piotr Findeisen <pi...@gmail.com>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* docs: add Ballista link to landing page (#17746) (#17775)
* docs: add Ballista link to landing page (#17746)
This adds a link and description for DataFusion Ballista to the landing page, as suggested in issue #17746. Ballista is a distributed compute platform built on top of DataFusion.
Closes: #17746
* fix(docs): update Ballista link
* updated theory part
* chore(deps): bump taiki-e/install-action from 2.62.6 to 2.62.8 (#17781)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.6 to 2.62.8.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/4575ae687efd0e2c78240087f26013fb2484987f...ea0eda622640ac23a17ba349cf09e2709d58f5e1)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.8
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump wasm-bindgen-test from 0.3.53 to 0.3.54 (#17784)
Bumps [wasm-bindgen-test](https://github.com/wasm-bindgen/wasm-bindgen) from 0.3.53 to 0.3.54.
- [Release notes](https://github.com/wasm-bindgen/wasm-bindgen/releases)
- [Changelog](https://github.com/wasm-bindgen/wasm-bindgen/blob/main/CHANGELOG.md)
- [Commits](https://github.com/wasm-bindgen/wasm-bindgen/commits)
---
updated-dependencies:
- dependency-name: wasm-bindgen-test
dependency-version: 0.3.54
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore: Action some old TODOs in github actions (#17694)
* chore: Action some old TODOs in github actions
* Update Cargo.toml
* testing
* Revert changing cli test runner to use container
* Remove sccache
* dev: Add benchmark for compilation profiles (#17754)
* Add benchmark for compilation profiles
* add apache header
* add apache header
* chore(deps): bump tokio-postgres from 0.7.13 to 0.7.14 (#17785)
Bumps [tokio-postgres](https://github.com/rust-postgres/rust-postgres) from 0.7.13 to 0.7.14.
- [Release notes](https://github.com/rust-postgres/rust-postgres/releases)
- [Commits](https://github.com/rust-postgres/rust-postgres/compare/tokio-postgres-v0.7.13...tokio-postgres-v0.7.14)
---
updated-dependencies:
- dependency-name: tokio-postgres
dependency-version: 0.7.14
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump serde from 1.0.226 to 1.0.227 (#17783)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.226 to 1.0.227.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.226...v1.0.227)
---
updated-dependencies:
- dependency-name: serde
dependency-version: 1.0.227
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump regex from 1.11.2 to 1.11.3 (#17782)
Bumps [regex](https://github.com/rust-lang/regex) from 1.11.2 to 1.11.3.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.11.2...1.11.3)
---
updated-dependencies:
- dependency-name: regex
dependency-version: 1.11.3
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Support `CAST` from temporal to `Utf8View` (#17535)
* Add case expr simplifiers for literal comparisons (#17743)
* Add case expr simplifiers for literal comparisons
* Update datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Avoid expr clones
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* chore: dependabot to run weekly (#17797)
* [DOCS] Add dbt Fusion engine and R2 Query Engine to "Known Users" (#17793)
* Add dbt Fusion engine and R2 Query Engine
* Update docs/source/user-guide/introduction.md
* Update docs/source/user-guide/introduction.md
* feat: change `datafusion-proto` to use `TaskContext` rather than`SessionContext` for physical plan serialization (#17601)
* change session context to task context in physical proto ...
* fix compilation issue
* remove `RuntimeEnv` from few function arguments
* update upgrading guide
* display window function's alias name in output (#17788)
* docs: update wasmtest README with instructions for Apple silicon (#17755)
* chore(deps): bump sysinfo from 0.37.0 to 0.37.1 (#17800)
Bumps [sysinfo](https://github.com/GuillaumeGomez/sysinfo) from 0.37.0 to 0.37.1.
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.37.0...v0.37.1)
---
updated-dependencies:
- dependency-name: sysinfo
dependency-version: 0.37.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump taiki-e/install-action from 2.62.8 to 2.62.9 (#17799)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.8 to 2.62.9.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/ea0eda622640ac23a17ba349cf09e2709d58f5e1...71d339ebf191fcbc3d49cd04b9484a4261f29975)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.9
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* feat(spark): implement Spark `make_dt_interval` function (#17728)
* feat(spark): implement Spark make_dt_interval function
* fmt
* delete pub
* test slt
* fmt
* overflow -> null
* sugested changes
* fmt
* only res in slt
* null not void type
* explain types
* explain types fix url
* better comment
* Fix potential overflow when we print verbose physical plan (#17798)
* change debug to trace for potential overflow
* fix comments.
* fix
* Add SedonaDB as known user to Apache DataFusion (#17806)
* Extend datatype semantic equality check to include timestamps (#17777)
* Extend datatype semantic equality to include timestamps
* test
* Respond to comments
* cargo fmt
---------
Co-authored-by: Shiv Bhatia <sb...@palantir.com>
* fix: Filter out nulls properly in approx_percentile_cont_with_weight (#17780)
* chore: refactor usage of `reassign_predicate_columns` (#17703)
* chore: refactor usage of `reassign_predicate_columns`
* chore: Address PR comments
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* dev: Add Apache license check to the lint script (#17787)
* Add liscense checker ci script
* fix the deliberately added bad license header
* review: use dev profile and pin the version
* Fix: common_sub_expression_eliminate optimizer rule failed (#16066)
Common_sub_expression_eliminate rule failed with error:
`SchemaError(FieldNotFound {field: <name>}, valid_fields: []})`
due to the schema being changed by the second application of
`find_common_exprs`
As I understood the source of the problem was in sequential call of
`find_common_exprs`. First call returned original names as `aggr_expr`
and changed names as `new_aggr_expr`. Second call takes into account
only `new_aggr_expr` and if names was already changed by first call
will return changed names as `aggr_expr`(original ones)
and put them into Projection logic.
I used NamePreserver mechanism to restore original schema names and
generate Projection with original name at the end of aggregate
optimization.
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* feat: support multi-threaded writing of Parquet files with modular encryption (#16738)
* Initial commit
diff --git c/Cargo.lock i/Cargo.lock
index 749971532..f0b9d0a5f 100644
--- c/Cargo.lock
+++ i/Cargo.lock
@@ -246,52 +246,62 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "arrow"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd798aea3553913a5986813e9c6ad31a2d2b04e931fe8ea4a37155eb541cebb5"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-arith",
- "arrow-array",
- "arrow-buffer",
- "arrow-cast",
+ "arrow-arith 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"arrow-csv",
- "arrow-data",
- "arrow-ipc",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"arrow-json",
- "arrow-ord",
+ "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"arrow-pyarrow",
- "arrow-row",
- "arrow-schema",
- "arrow-select",
- "arrow-string",
+ "arrow-row 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-string 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"half",
"rand 0.9.2",
]
[[package]]
name = "arrow-arith"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "508dafb53e5804a238cab7fd97a59ddcbfab20cc4d9814b1ab5465b9fa147f2e"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "chrono",
+ "num",
+]
+
+[[package]]
+name = "arrow-arith"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"chrono",
"num",
]
[[package]]
name = "arrow-array"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e2730bc045d62bb2e53ef8395b7d4242f5c8102f41ceac15e8395b9ac3d08461"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
"ahash 0.8.12",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"chrono",
"chrono-tz",
"half",
@@ -299,11 +309,35 @@ dependencies = [
"num",
]
+[[package]]
+name = "arrow-array"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "ahash 0.8.12",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "chrono",
+ "half",
+ "hashbrown 0.15.4",
+ "num",
+]
+
[[package]]
name = "arrow-buffer"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "54295b93beb702ee9a6f6fbced08ad7f4d76ec1c297952d4b83cf68755421d1d"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+dependencies = [
+ "bytes",
+ "half",
+ "num",
+]
+
+[[package]]
+name = "arrow-buffer"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
dependencies = [
"bytes",
"half",
@@ -312,15 +346,14 @@ dependencies = [
[[package]]
name = "arrow-cast"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "67e8bcb7dc971d779a7280593a1bf0c2743533b8028909073e804552e85e75b5"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "arrow-select",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"atoi",
"base64 0.22.1",
"chrono",
@@ -332,14 +365,32 @@ dependencies = [
]
[[package]]
-name = "arrow-csv"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "673fd2b5fb57a1754fdbfac425efd7cf54c947ac9950c1cce86b14e248f1c458"
+name = "arrow-cast"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
dependencies = [
- "arrow-array",
- "arrow-cast",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "atoi",
+ "base64 0.22.1",
+ "chrono",
+ "half",
+ "lexical-core",
+ "num",
+ "ryu",
+]
+
+[[package]]
+name = "arrow-csv"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"chrono",
"csv",
"csv-core",
@@ -348,33 +399,42 @@ dependencies = [
[[package]]
name = "arrow-data"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "97c22fe3da840039c69e9f61f81e78092ea36d57037b4900151f063615a2f6b4"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-buffer",
- "arrow-schema",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "half",
+ "num",
+]
+
+[[package]]
+name = "arrow-data"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"half",
"num",
]
[[package]]
name = "arrow-flight"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6808d235786b721e49e228c44dd94242f2e8b46b7e95b233b0733c46e758bfee"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
dependencies = [
- "arrow-arith",
- "arrow-array",
- "arrow-buffer",
- "arrow-cast",
- "arrow-data",
- "arrow-ipc",
- "arrow-ord",
- "arrow-row",
- "arrow-schema",
- "arrow-select",
- "arrow-string",
+ "arrow-arith 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-row 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-string 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"base64 0.22.1",
"bytes",
"futures",
@@ -382,35 +442,45 @@ dependencies = [
"paste",
"prost",
"prost-types",
- "tonic",
+ "tonic 0.12.3",
]
[[package]]
name = "arrow-ipc"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "778de14c5a69aedb27359e3dd06dd5f9c481d5f6ee9fbae912dba332fd64636b"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"flatbuffers",
"lz4_flex",
"zstd",
]
[[package]]
-name = "arrow-json"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3860db334fe7b19fcf81f6b56f8d9d95053f3839ffe443d56b5436f7a29a1794"
+name = "arrow-ipc"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-cast",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "flatbuffers",
+]
+
+[[package]]
+name = "arrow-json"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"chrono",
"half",
"indexmap 2.10.0",
@@ -424,78 +494,130 @@ dependencies = [
[[package]]
name = "arrow-ord"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "425fa0b42a39d3ff55160832e7c25553e7f012c3f187def3d70313e7a29ba5d9"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "arrow-select",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+]
+
+[[package]]
+name = "arrow-ord"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
]
[[package]]
name = "arrow-pyarrow"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d944d8ae9b77230124e6570865b570416c33a5809f32c4136c679bbe774e45c9"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"pyo3",
]
[[package]]
name = "arrow-row"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "df9c9423c9e71abd1b08a7f788fcd203ba2698ac8e72a1f236f1faa1a06a7414"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "half",
+]
+
+[[package]]
+name = "arrow-row"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"half",
]
[[package]]
name = "arrow-schema"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "85fa1babc4a45fdc64a92175ef51ff00eba5ebbc0007962fecf8022ac1c6ce28"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
"bitflags 2.9.1",
"serde",
"serde_json",
]
+[[package]]
+name = "arrow-schema"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+
[[package]]
name = "arrow-select"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d8854d15f1cf5005b4b358abeb60adea17091ff5bdd094dca5d3f73787d81170"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
"ahash 0.8.12",
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "num",
+]
+
+[[package]]
+name = "arrow-select"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "ahash 0.8.12",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"num",
]
[[package]]
name = "arrow-string"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c477e8b89e1213d5927a2a84a72c384a9bf4dd0dbf15f9fd66d821aafd9e95e"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "arrow-select",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "memchr",
+ "num",
+ "regex",
+ "regex-syntax",
+]
+
+[[package]]
+name = "arrow-string"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+dependencies = [
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
"memchr",
"num",
"regex",
@@ -567,6 +689,28 @@ dependencies = [
"syn 2.0.106",
]
+[[package]]
+name = "async-stream"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476"
+dependencies = [
+ "async-stream-impl",
+ "futures-core",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "async-stream-impl"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.104",
+]
+
[[package]]
name = "async-trait"
version = "0.1.89"
@@ -827,7 +971,7 @@ dependencies = [
"rustls-native-certs",
"rustls-pki-types",
"tokio",
- "tower",
+ "tower 0.5.2",
"tracing",
]
@@ -948,18 +1092,19 @@ dependencies = [
[[package]]
name = "axum"
-version = "0.8.4"
+version = "0.7.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "021e862c184ae977658b36c4500f7feac3221ca5da43e3f25bd04ab6c79a29b5"
+checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f"
dependencies = [
- "axum-core",
+ "async-trait",
+ "axum-core 0.4.5",
"bytes",
"futures-util",
"http 1.3.1",
"http-body 1.0.1",
"http-body-util",
"itoa",
- "matchit",
+ "matchit 0.7.3",
"memchr",
"mime",
"percent-encoding",
@@ -967,7 +1112,53 @@ dependencies = [
"rustversion",
"serde",
"sync_wrapper",
- "tower",
+ "tower 0.5.2",
+ "tower-layer",
+ "tower-service",
+]
+
+[[package]]
+name = "axum"
+version = "0.8.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "021e862c184ae977658b36c4500f7feac3221ca5da43e3f25bd04ab6c79a29b5"
+dependencies = [
+ "axum-core 0.5.2",
+ "bytes",
+ "futures-util",
+ "http 1.3.1",
+ "http-body 1.0.1",
+ "http-body-util",
+ "itoa",
+ "matchit 0.8.4",
+ "memchr",
+ "mime",
+ "percent-encoding",
+ "pin-project-lite",
+ "rustversion",
+ "serde",
+ "sync_wrapper",
+ "tower 0.5.2",
+ "tower-layer",
+ "tower-service",
+]
+
+[[package]]
+name = "axum-core"
+version = "0.4.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199"
+dependencies = [
+ "async-trait",
+ "bytes",
+ "futures-util",
+ "http 1.3.1",
+ "http-body 1.0.1",
+ "http-body-util",
+ "mime",
+ "pin-project-lite",
+ "rustversion",
+ "sync_wrapper",
"tower-layer",
"tower-service",
]
@@ -1818,8 +2009,8 @@ name = "datafusion"
version = "49.0.1"
dependencies = [
"arrow",
- "arrow-ipc",
- "arrow-schema",
+ "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"async-trait",
"bytes",
"bzip2 0.6.0",
@@ -1996,7 +2187,7 @@ dependencies = [
"ahash 0.8.12",
"apache-avro",
"arrow",
- "arrow-ipc",
+ "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"base64 0.22.1",
"chrono",
"half",
@@ -2176,7 +2367,7 @@ version = "49.0.1"
dependencies = [
"arrow",
"arrow-flight",
- "arrow-schema",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"async-trait",
"base64 0.22.1",
"bytes",
@@ -2197,7 +2388,7 @@ dependencies = [
"tempfile",
"test-utils",
"tokio",
- "tonic",
+ "tonic 0.13.1",
"tracing",
"tracing-subscriber",
"url",
@@ -2264,7 +2455,7 @@ version = "49.0.1"
dependencies = [
"abi_stable",
"arrow",
- "arrow-schema",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"async-ffi",
"async-trait",
"datafusion",
@@ -2284,7 +2475,7 @@ name = "datafusion-functions"
version = "49.0.1"
dependencies = [
"arrow",
- "arrow-buffer",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"base64 0.22.1",
"blake2",
"blake3",
@@ -2347,7 +2538,7 @@ name = "datafusion-functions-nested"
version = "49.0.1"
dependencies = [
"arrow",
- "arrow-ord",
+ "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"criterion",
"datafusion-common",
"datafusion-doc",
@@ -2517,8 +2708,8 @@ version = "49.0.1"
dependencies = [
"ahash 0.8.12",
"arrow",
- "arrow-ord",
- "arrow-schema",
+ "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"async-trait",
"chrono",
"criterion",
@@ -2589,7 +2780,7 @@ name = "datafusion-pruning"
version = "49.0.1"
dependencies = [
"arrow",
- "arrow-schema",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"datafusion-common",
"datafusion-datasource",
"datafusion-expr",
@@ -4157,6 +4348,12 @@ dependencies = [
"pkg-config",
]
+[[package]]
+name = "matchit"
+version = "0.7.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
+
[[package]]
name = "matchit"
version = "0.8.4"
@@ -4529,18 +4726,17 @@ dependencies = [
[[package]]
name = "parquet"
-version = "56.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c7288a07ed5d25939a90f9cb1ca5afa6855faa08ec7700613511ae64bdb0620c"
+version = "55.2.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
dependencies = [
"ahash 0.8.12",
- "arrow-array",
- "arrow-buffer",
- "arrow-cast",
- "arrow-data",
- "arrow-ipc",
- "arrow-schema",
- "arrow-select",
+ "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
"base64 0.22.1",
"brotli",
"bytes",
@@ -5449,7 +5645,7 @@ dependencies = [
"tokio",
"tokio-rustls",
"tokio-util",
- "tower",
+ "tower 0.5.2",
"tower-http",
"tower-service",
"url",
@@ -6681,12 +6877,13 @@ dependencies = [
[[package]]
name = "tonic"
-version = "0.13.1"
+version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e581ba15a835f4d9ea06c55ab1bd4dce26fc53752c69a04aac00703bfb49ba9"
+checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52"
dependencies = [
+ "async-stream",
"async-trait",
- "axum",
+ "axum 0.7.9",
"base64 0.22.1",
"bytes",
"h2",
@@ -6702,7 +6899,56 @@ dependencies = [
"socket2 0.5.10",
"tokio",
"tokio-stream",
- "tower",
+ "tower 0.4.13",
+ "tower-layer",
+ "tower-service",
+ "tracing",
+]
+
+[[package]]
+name = "tonic"
+version = "0.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7e581ba15a835f4d9ea06c55ab1bd4dce26fc53752c69a04aac00703bfb49ba9"
+dependencies = [
+ "async-trait",
+ "axum 0.8.4",
+ "base64 0.22.1",
+ "bytes",
+ "h2",
+ "http 1.3.1",
+ "http-body 1.0.1",
+ "http-body-util",
+ "hyper",
+ "hyper-timeout",
+ "hyper-util",
+ "percent-encoding",
+ "pin-project",
+ "prost",
+ "socket2 0.5.10",
+ "tokio",
+ "tokio-stream",
+ "tower 0.5.2",
+ "tower-layer",
+ "tower-service",
+ "tracing",
+]
+
+[[package]]
+name = "tower"
+version = "0.4.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
+dependencies = [
+ "futures-core",
+ "futures-util",
+ "indexmap 1.9.3",
+ "pin-project",
+ "pin-project-lite",
+ "rand 0.8.5",
+ "slab",
+ "tokio",
+ "tokio-util",
"tower-layer",
"tower-service",
"tracing",
@@ -6740,7 +6986,7 @@ dependencies = [
"http-body 1.0.1",
"iri-string",
"pin-project-lite",
- "tower",
+ "tower 0.5.2",
"tower-layer",
"tower-service",
]
diff --git c/Cargo.toml i/Cargo.toml
index 5915035cd..5ee3cc566 100644
--- c/Cargo.toml
+++ i/Cargo.toml
@@ -90,19 +90,20 @@ ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
apache-avro = { version = "0.17", default-features = false }
-arrow = { version = "56.0.0", features = [
+arrow = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", features = [
"prettyprint",
"chrono-tz",
] }
-arrow-buffer = { version = "56.0.0", default-features = false }
-arrow-flight = { version = "56.0.0", features = [
+
+arrow-buffer = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", default-features = false }
+arrow-flight = { git = "https://github.com/rok/arrow-rs.git", features = [
"flight-sql-experimental",
] }
-arrow-ipc = { version = "56.0.0", default-features = false, features = [
+arrow-ipc = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", default-features = false, features = [
"lz4",
] }
-arrow-ord = { version = "56.0.0", default-features = false }
-arrow-schema = { version = "56.0.0", default-features = false }
+arrow-ord = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", default-features = false }
+arrow-schema = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", default-features = false }
async-trait = "0.1.89"
bigdecimal = "0.4.8"
bytes = "1.10"
@@ -157,7 +158,7 @@ itertools = "0.14"
log = "^0.4"
object_store = { version = "0.12.3", default-features = false }
parking_lot = "0.12"
-parquet = { version = "56.0.0", default-features = false, features = [
+parquet = { git = "https://github.com/rok/arrow-rs.git", branch = "multi-threaded_encrypted_writing", default-features = false, features = [
"arrow",
"async",
"object_store",
diff --git c/datafusion-examples/Cargo.toml i/datafusion-examples/Cargo.toml
index f12bd9202..b4c8d3507 100644
--- c/datafusion-examples/Cargo.toml
+++ i/datafusion-examples/Cargo.toml
@@ -32,18 +32,6 @@ rust-version = { workspace = true }
[lints]
workspace = true
-[[example]]
-name = "flight_sql_server"
-path = "examples/flight/flight_sql_server.rs"
-
-[[example]]
-name = "flight_server"
-path = "examples/flight/flight_server.rs"
-
-[[example]]
-name = "flight_client"
-path = "examples/flight/flight_client.rs"
-
[[example]]
name = "dataframe_to_s3"
path = "examples/external_dependency/dataframe-to-s3.rs"
diff --git c/datafusion/common/Cargo.toml i/datafusion/common/Cargo.toml
index afd74c7be..8040b3ad1 100644
--- c/datafusion/common/Cargo.toml
+++ i/datafusion/common/Cargo.toml
@@ -71,7 +71,7 @@ log = { workspace = true }
object_store = { workspace = true, optional = true }
parquet = { workspace = true, optional = true, default-features = true }
paste = "1.0.15"
-pyo3 = { version = "0.25", optional = true }
+pyo3 = { version = "0.25.1", optional = true }
recursive = { workspace = true, optional = true }
sqlparser = { workspace = true }
tokio = { workspace = true }
diff --git c/datafusion/common/src/file_options/parquet_writer.rs i/datafusion/common/src/file_options/parquet_writer.rs
index 185826aef..d7b490af0 100644
--- c/datafusion/common/src/file_options/parquet_writer.rs
+++ i/datafusion/common/src/file_options/parquet_writer.rs
@@ -25,6 +25,8 @@ use crate::{
DataFusionError, Result, _internal_datafusion_err,
};
+pub const DEFAULT_MAX_STATISTICS_SIZE: usize = 4096;
+
use arrow::datatypes::Schema;
// TODO: handle once deprecated
#[allow(deprecated)]
diff --git c/datafusion/common/src/scalar/mod.rs i/datafusion/common/src/scalar/mod.rs
index 51247612e..8f8c52086 100644
--- c/datafusion/common/src/scalar/mod.rs
+++ i/datafusion/common/src/scalar/mod.rs
@@ -2386,7 +2386,9 @@ impl ScalarValue {
| DataType::Time64(TimeUnit::Millisecond)
| DataType::RunEndEncoded(_, _)
| DataType::ListView(_)
- | DataType::LargeListView(_) => {
+ | DataType::LargeListView(_)
+ | DataType::Decimal32(_, _)
+ | DataType::Decimal64(_, _) => {
return _not_impl_err!(
"Unsupported creation of {:?} array from ScalarValue {:?}",
data_type,
diff --git c/datafusion/core/src/dataframe/parquet.rs i/datafusion/core/src/dataframe/parquet.rs
index 83bb60184..01149c1ec 100644
--- c/datafusion/core/src/dataframe/parquet.rs
+++ i/datafusion/core/src/dataframe/parquet.rs
@@ -278,6 +278,7 @@ mod tests {
// Write encrypted parquet using write_parquet
let mut options = TableParquetOptions::default();
options.crypto.file_encryption = Some((&encrypt).into());
+ options.global.allow_single_file_parallelism = true;
df.write_parquet(
tempfile_str.as_str(),
diff --git c/datafusion/core/tests/fuzz_cases/pruning.rs i/datafusion/core/tests/fuzz_cases/pruning.rs
index c6e30c072..4ab1f08f1 100644
--- c/datafusion/core/tests/fuzz_cases/pruning.rs
+++ i/datafusion/core/tests/fuzz_cases/pruning.rs
@@ -314,7 +314,7 @@ async fn execute_with_predicate(
}
async fn write_parquet_file(
- truncation_length: Option<usize>,
+ _truncation_length: Option<usize>,
schema: Arc<Schema>,
row_groups: Vec<Vec<String>>,
) -> Bytes {
diff --git c/datafusion/datasource-avro/src/avro_to_arrow/schema.rs i/datafusion/datasource-avro/src/avro_to_arrow/schema.rs
index cc87d3c1c..00b3f9d6d 100644
--- c/datafusion/datasource-avro/src/avro_to_arrow/schema.rs
+++ i/datafusion/datasource-avro/src/avro_to_arrow/schema.rs
@@ -239,6 +239,8 @@ fn default_field_name(dt: &DataType) -> &str {
DataType::Decimal64(_, _) => "decimal",
DataType::Decimal128(_, _) => "decimal",
DataType::Decimal256(_, _) => "decimal",
+ DataType::Decimal32(_, _) => "decimal",
+ DataType::Decimal64(_, _) => "decimal",
}
}
diff --git c/datafusion/datasource-parquet/src/file_format.rs i/datafusion/datasource-parquet/src/file_format.rs
index 56718534a..934a7b2ee 100644
--- c/datafusion/datasource-parquet/src/file_format.rs
+++ i/datafusion/datasource-parquet/src/file_format.rs
@@ -78,8 +78,8 @@ use object_store::path::Path;
use object_store::{ObjectMeta, ObjectStore};
use parquet::arrow::arrow_reader::statistics::StatisticsConverter;
use parquet::arrow::arrow_writer::{
- compute_leaves, get_column_writers, ArrowColumnChunk, ArrowColumnWriter,
- ArrowLeafColumn, ArrowWriterOptions,
+ compute_leaves, ArrowColumnChunk, ArrowColumnWriter, ArrowLeafColumn,
+ ArrowRowGroupWriterFactory, ArrowWriterOptions,
};
use parquet::arrow::async_reader::MetadataFetch;
use parquet::arrow::{parquet_to_arrow_schema, ArrowSchemaConverter, AsyncArrowWriter};
@@ -1570,7 +1570,7 @@ impl FileSink for ParquetSink {
while let Some((path, mut rx)) = file_stream_rx.recv().await {
let parquet_props = self.create_writer_props(&runtime, &path)?;
- if !allow_single_file_parallelism {
+ if !parquet_opts.global.allow_single_file_parallelism {
let mut writer = self
.create_async_arrow_writer(
&path,
@@ -1698,13 +1698,13 @@ type ColSender = Sender<ArrowLeafColumn>;
/// Returns join handles for each columns serialization task along with a send channel
/// to send arrow arrays to each serialization task.
fn spawn_column_parallel_row_group_writer(
- schema: Arc<Schema>,
- parquet_props: Arc<WriterProperties>,
+ arrow_row_group_writer_factory: Arc<ArrowRowGroupWriterFactory>,
max_buffer_size: usize,
pool: &Arc<dyn MemoryPool>,
) -> Result<(Vec<ColumnWriterTask>, Vec<ColSender>)> {
- let schema_desc = ArrowSchemaConverter::new().convert(&schema)?;
- let col_writers = get_column_writers(&schema_desc, &parquet_props, &schema)?;
+ let arrow_row_group_writer =
+ arrow_row_group_writer_factory.create_row_group_writer(0)?;
+ let col_writers = arrow_row_group_writer.into_column_writers();
let num_columns = col_writers.len();
let mut col_writer_tasks = Vec::with_capacity(num_columns);
@@ -1799,6 +1799,7 @@ fn spawn_rg_join_and_finalize_task(
/// across both columns and row_groups, with a theoretical max number of parallel tasks
/// given by n_columns * num_row_groups.
fn spawn_parquet_parallel_serialization_task(
+ arrow_row_group_writer_factory: Arc<ArrowRowGroupWriterFactory>,
mut data: Receiver<RecordBatch>,
serialize_tx: Sender<SpawnedTask<RBStreamSerializeResult>>,
schema: Arc<Schema>,
@@ -1811,12 +1812,14 @@ fn spawn_parquet_parallel_serialization_task(
let max_row_group_rows = writer_props.max_row_group_size();
let (mut column_writer_handles, mut col_array_channels) =
spawn_column_parallel_row_group_writer(
- Arc::clone(&schema),
- Arc::clone(&writer_props),
+ Arc::clone(&arrow_row_group_writer_factory),
max_buffer_rb,
&pool,
)?;
let mut current_rg_rows = 0;
+ // TODO: row_group_writer should use the correct row group index. Currently this would fail if
+ // multiple row groups were written.
+ // let mut rg_index = 0;
while let Some(mut rb) = data.recv().await {
// This loop allows the "else" block to repeatedly split the RecordBatch to handle the case
@@ -1863,8 +1866,7 @@ fn spawn_parquet_parallel_serialization_task(
(column_writer_handles, col_array_channels) =
spawn_column_parallel_row_group_writer(
- Arc::clone(&schema),
- Arc::clone(&writer_props),
+ Arc::clone(&arrow_row_group_writer_factory),
max_buffer_rb,
&pool,
)?;
@@ -1895,24 +1897,15 @@ fn spawn_parquet_parallel_serialization_task(
/// Consume RowGroups serialized by other parallel tasks and concatenate them in
/// to the final parquet file, while flushing finalized bytes to an [ObjectStore]
async fn concatenate_parallel_row_groups(
+ mut parquet_writer: SerializedFileWriter<SharedBuffer>,
+ merged_buff: SharedBuffer,
mut serialize_rx: Receiver<SpawnedTask<RBStreamSerializeResult>>,
- schema: Arc<Schema>,
- writer_props: Arc<WriterProperties>,
mut object_store_writer: Box<dyn AsyncWrite + Send + Unpin>,
pool: Arc<dyn MemoryPool>,
) -> Result<FileMetaData> {
- let merged_buff = SharedBuffer::new(INITIAL_BUFFER_BYTES);
-
let mut file_reservation =
MemoryConsumer::new("ParquetSink(SerializedFileWriter)").register(&pool);
- let schema_desc = ArrowSchemaConverter::new().convert(schema.as_ref())?;
- let mut parquet_writer = SerializedFileWriter::new(
- merged_buff.clone(),
- schema_desc.root_schema_ptr(),
- writer_props,
- )?;
-
while let Some(task) = serialize_rx.recv().await {
let result = task.join_unwind().await;
let mut rg_out = parquet_writer.next_row_group()?;
@@ -1963,8 +1956,25 @@ async fn output_single_parquet_file_parallelized(
let (serialize_tx, serialize_rx) =
mpsc::channel::<SpawnedTask<RBStreamSerializeResult>>(max_rowgroups);
+ let parquet_schema = ArrowSchemaConverter::new()
+ .with_coerce_types(parquet_props.coerce_types())
+ .convert(&output_schema)?;
+ let merged_buff = SharedBuffer::new(INITIAL_BUFFER_BYTES);
+ let parquet_writer = SerializedFileWriter::new(
+ merged_buff.clone(),
+ parquet_schema.root_schema_ptr(),
+ parquet_props.clone().into(),
+ )?;
+ let arrow_row_group_writer_factory = ArrowRowGroupWriterFactory::new(
+ &parquet_writer,
+ parquet_schema,
+ Arc::clone(&output_schema),
+ parquet_props.clone().into(),
+ );
+
let arc_props = Arc::new(parquet_props.clone());
let launch_serialization_task = spawn_parquet_parallel_serialization_task(
+ Arc::new(arrow_row_group_writer_factory),
data,
serialize_tx,
Arc::clone(&output_schema),
@@ -1972,19 +1982,21 @@ async fn output_single_parquet_file_parallelized(
parallel_options,
Arc::clone(&pool),
);
- let file_metadata = concatenate_parallel_row_groups(
- serialize_rx,
- Arc::clone(&output_schema),
- Arc::clone(&arc_props),
- object_store_writer,
- pool,
- )
- .await?;
launch_serialization_task
.join_unwind()
.await
.map_err(|e| DataFusionError::ExecutionJoin(Box::new(e)))??;
+
+ let file_metadata = concatenate_parallel_row_groups(
+ parquet_writer,
+ merged_buff,
+ serialize_rx,
+ object_store_writer,
+ pool,
+ )
+ .await?;
+
Ok(file_metadata)
}
diff --git c/datafusion/expr/src/utils.rs i/datafusion/expr/src/utils.rs
index 7a612b6fe..cd8e419ac 100644
--- c/datafusion/expr/src/utils.rs
+++ i/datafusion/expr/src/utils.rs
@@ -818,6 +818,8 @@ pub fn can_hash(data_type: &DataType) -> bool {
DataType::Decimal64(_, _) => true,
DataType::Decimal128(_, _) => true,
DataType::Decimal256(_, _) => true,
+ DataType::Decimal32(_, _) => true,
+ DataType::Decimal64(_, _) => true,
DataType::Timestamp(_, _) => true,
DataType::Utf8 => true,
DataType::LargeUtf8 => true,
diff --git c/datafusion/sql/src/unparser/expr.rs i/datafusion/sql/src/unparser/expr.rs
index 0501a4e04..86c648cba 100644
--- c/datafusion/sql/src/unparser/expr.rs
+++ i/datafusion/sql/src/unparser/expr.rs
@@ -1729,7 +1729,9 @@ impl Unparser<'_> {
not_impl_err!("Unsupported DataType: conversion: {data_type:?}")
}
DataType::Decimal128(precision, scale)
- | DataType::Decimal256(precision, scale) => {
+ | DataType::Decimal256(precision, scale)
+ | DataType::Decimal32(precision, scale)
+ | DataType::Decimal64(precision, scale) => {
let mut new_precision = *precision as u64;
let mut new_scale = *scale as u64;
if *scale < 0 {
diff --git c/datafusion/sqllogictest/test_files/copy.slt i/datafusion/sqllogictest/test_files/copy.slt
index 096cde86f..e16fcfe84 100644
--- c/datafusion/sqllogictest/test_files/copy.slt
+++ i/datafusion/sqllogictest/test_files/copy.slt
@@ -306,7 +306,7 @@ select * from validate_struct_with_array;
# Copy parquet with all supported statement overrides
-query I
+query error DataFusion error: Invalid or Unsupported Configuration: Config value "max_statistics_size" not found on ParquetOptions
COPY source_table
TO 'test_files/scratch/copy/table_with_options/'
STORED AS PARQUET
@@ -336,8 +336,6 @@ OPTIONS (
'format.bloom_filter_ndv' 100,
'format.metadata::key' 'value'
)
-----
-2
# valid vs invalid metadata
@@ -404,11 +402,8 @@ OPTIONS (
statement ok
CREATE EXTERNAL TABLE validate_parquet_with_options STORED AS PARQUET LOCATION 'test_files/scratch/copy/table_with_options/';
-query IT
+statement count 0
select * from validate_parquet_with_options;
-----
-1 Foo
-2 Bar
# Copy from table to single file
query I
diff --git c/datafusion/substrait/src/logical_plan/consumer/utils.rs i/datafusion/substrait/src/logical_plan/consumer/utils.rs
index f7eedcb7a..f809bc82a 100644
--- c/datafusion/substrait/src/logical_plan/consumer/utils.rs
+++ i/datafusion/substrait/src/logical_plan/consumer/utils.rs
@@ -216,7 +216,9 @@ pub fn rename_data_type(
| DataType::Decimal32(_, _)
| DataType::Decimal64(_, _)
| DataType::Decimal128(_, _)
- | DataType::Decimal256(_, _) => Ok(data_type.clone()),
+ | DataType::Decimal256(_, _)
+ | DataType::Decimal32(_, _)
+ | DataType::Decimal64(_, _) => Ok(data_type.clone()),
}
}
* Rebase fixes, switch to ArrowWriter approach
diff --git c/Cargo.lock i/Cargo.lock
index f0b9d0a5f..373239aab 100644
--- c/Cargo.lock
+++ i/Cargo.lock
@@ -246,62 +246,62 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "arrow"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-arith 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-arith 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-cast 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"arrow-csv",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-ipc 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"arrow-json",
- "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-ord 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"arrow-pyarrow",
- "arrow-row 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-string 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-row 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-string 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"half",
"rand 0.9.2",
]
[[package]]
name = "arrow-arith"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"chrono",
"num",
]
[[package]]
name = "arrow-arith"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"chrono",
"num",
]
[[package]]
name = "arrow-array"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
"ahash 0.8.12",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"chrono",
"chrono-tz",
"half",
@@ -311,13 +311,13 @@ dependencies = [
[[package]]
name = "arrow-array"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
"ahash 0.8.12",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"chrono",
"half",
"hashbrown 0.15.4",
@@ -326,8 +326,8 @@ dependencies = [
[[package]]
name = "arrow-buffer"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
"bytes",
"half",
@@ -336,8 +336,8 @@ dependencies = [
[[package]]
name = "arrow-buffer"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
"bytes",
"half",
@@ -346,14 +346,14 @@ dependencies = [
[[package]]
name = "arrow-cast"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"atoi",
"base64 0.22.1",
"chrono",
@@ -366,14 +366,14 @@ dependencies = [
[[package]]
name = "arrow-cast"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"atoi",
"base64 0.22.1",
"chrono",
@@ -385,12 +385,12 @@ dependencies = [
[[package]]
name = "arrow-csv"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-cast 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"chrono",
"csv",
"csv-core",
@@ -399,42 +399,42 @@ dependencies = [
[[package]]
name = "arrow-data"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"half",
"num",
]
[[package]]
name = "arrow-data"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"half",
"num",
]
[[package]]
name = "arrow-flight"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-arith 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-ipc 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-ord 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-row 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-string 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-arith 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-cast 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-ipc 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-ord 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-row 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-string 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"base64 0.22.1",
"bytes",
"futures",
@@ -442,18 +442,18 @@ dependencies = [
"paste",
"prost",
"prost-types",
- "tonic 0.12.3",
+ "tonic",
]
[[package]]
name = "arrow-ipc"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"flatbuffers",
"lz4_flex",
"zstd",
@@ -461,26 +461,26 @@ dependencies = [
[[package]]
name = "arrow-ipc"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
"flatbuffers",
]
[[package]]
name = "arrow-json"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-cast 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-cast 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
"chrono",
"half",
"indexmap 2.10.0",
@@ -494,67 +494,67 @@ dependencies = [
[[package]]
name = "arrow-ord"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2)",
]
[[package]]
name = "arrow-ord"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git#674dc17b2c423be16d0725a6537b0063ac7b1b58"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git#876585c1cd986dbaee0c26d52b55a4186a2f68c8"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-buffer 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-schema 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
- "arrow-select 55.2.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-array 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-buffer 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-data 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-schema 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
+ "arrow-select 56.0.0 (git+https://github.com/rok/arrow-rs.git)",
]
[[package]]
name = "arrow-pyarrow"
-version = "55.2.0"
-source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing#b9396ccee27a39c91feccc982f5e976f0c0ff6d8"
+version = "56.0.0"
+source = "git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing_2#d9590212db94de291203220e2ed0beb808c69072"
dependencies = [
- "arrow-array 55.2.0 (git+https://github.com/rok/arrow-rs.git?branch=multi-threaded_encrypted_writing)",
- "arrow-data 55.2.0 (git+https://github.com/rok/arrow-rs.git?bra…
* feat(spark): implement Spark `map` function `map_from_entries` (#17779)
* feat(spark): implement Spark `map` function `map_from_entries`
* fix: map_from_entries with null entries in lists, chore: refactor initial offsets, add tests
* chore: remove dialect settings in SLT tests that are outdated (#17807)
* feat: Add Hash Join benchmarks (#17636)
* feat: Add Hash Join benchmarks
* fmt
* Update benchmarks/README.md
Co-authored-by: Adrian Garcia Badaracco <17...@users.noreply.github.com>
* add benchmarks
* update selectivities
* fix the error introduced when merging main
---------
Co-authored-by: Adrian Garcia Badaracco <17...@users.noreply.github.com>
Co-authored-by: Yongting You <20...@gmail.com>
* chore(deps): bump thiserror from 2.0.16 to 2.0.17 (#17821)
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 2.0.16 to 2.0.17.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.16...2.0.17)
---
updated-dependencies:
- dependency-name: thiserror
dependency-version: 2.0.17
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump quote from 1.0.40 to 1.0.41 (#17822)
Bumps [quote](https://github.com/dtolnay/quote) from 1.0.40 to 1.0.41.
- [Release notes](https://github.com/dtolnay/quote/releases)
- [Commits](https://github.com/dtolnay/quote/compare/1.0.40...1.0.41)
---
updated-dependencies:
- dependency-name: quote
dependency-version: 1.0.41
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump taiki-e/install-action from 2.62.9 to 2.62.12 (#17823)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.9 to 2.62.12.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/71d339ebf191fcbc3d49cd04b9484a4261f29975...5ab30948b991e8d6aa5a6c1e33c6aea130c6de65)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.12
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* chore(deps): bump serde from 1.0.227 to 1.0.228 (#17827)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.227 to 1.0.228.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.227...v1.0.228)
---
updated-dependencies:
- dependency-name: serde
dependency-version: 1.0.228
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
* Disable failing benchmark query (#17809)
* chore(deps): bump taiki-e/install-action from 2.62.12 to 2.62.13 (#17836)
Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.62.12 to 2.62.13.
- [Release notes](https://github.com/taiki-e/install-action/releases)
- [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/install-action/compare/5ab30948b991e8d6aa5a6c1e33c6aea130c6de65...d0f4f69b07c0804d1003ca9a5a5f853423872ed9)
---
updated-dependencies:
- dependency-name: taiki-e/install-action
dependency-version: 2.62.13
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <su...@github.com>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
---------
Signed-off-by: dependabot[bot] <su...@github.com>
Signed-off-by: Dennis Zhuang <ki...@gmail.com>
Co-authored-by: Emil Ernerfeldt <em...@gmail.com>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
Co-authored-by: Jonathan Chen <ch...@gmail.com>
Co-authored-by: Jeffrey Vo <je...@gmail.com>
Co-authored-by: YuNing Chen <ad...@ynchen.me>
Co-authored-by: dependabot[bot] <49...@users.noreply.github.com>
Co-authored-by: ding-young <ls...@snu.ac.kr>
Co-authored-by: Filippo Rossi <12...@users.noreply.github.com>
Co-authored-by: Yongting You <20...@gmail.com>
Co-authored-by: Nga Tran <ng...@live.com>
Co-authored-by: Kristin Cowalcijk <bo...@wherobots.com>
Co-authored-by: Rohan Krishnaswamy <47...@users.noreply.github.com>
Co-authored-by: Jack Kleeman <ja...@gmail.com>
Co-authored-by: Bruno Volpato <br...@gmail.com>
Co-authored-by: Dan Lovell <dl...@gmail.com>
Co-authored-by: Wes McKinney <we...@apache.org>
Co-authored-by: Adrian Garcia Badaracco <17...@users.noreply.github.com>
Co-authored-by: Mustafa Akur <ak...@gmail.com>
Co-authored-by: Qi Zhu <82...@qq.com>
Co-authored-by: 张林伟 <le...@gmail.com>
Co-authored-by: xudong.w <wx...@gmail.com>
Co-authored-by: Oleks V <co...@users.noreply.github.com>
Co-authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Co-authored-by: Alex Huang <hu...@gmail.com>
Co-authored-by: Yang Jiang <ji...@163.com>
Co-authored-by: Yijie Shen <he...@gmail.com>
Co-authored-by: Brent Gardner <bg...@squarelabs.net>
Co-authored-by: Jax Liu <li...@gmail.com>
Co-authored-by: Ifeanyi Ubah <if...@yahoo.com>
Co-authored-by: Will Jones <wi...@gmail.com>
Co-authored-by: Paddy Horan <57...@users.noreply.github.com>
Co-authored-by: Dan Harris <13...@users.noreply.github.com>
Co-authored-by: Ruihang Xia <wa...@gmail.com>
Co-authored-by: Parth Chandra <pa...@apache.org>
Co-authored-by: Adam Gutglick <ad...@spiraldb.com>
Co-authored-by: Viktor Yershov <vi...@spice.ai>
Co-authored-by: Bruno Volpato <br...@datadoghq.com>
Co-authored-by: Chen Chongchen <ch...@qq.com>
Co-authored-by: Duong Cong Toai <35...@users.noreply.github.com>
Co-authored-by: David López <ho...@devel0pez.com>
Co-authored-by: Nuno Faria <nu...@gmail.com>
Co-authored-by: Qi Zhu <qi...@polygon.io>
Co-authored-by: Marko Milenković <mi...@users.noreply.github.com>
Co-authored-by: Peter Nguyen <pe...@gmail.com>
Co-authored-by: Piotr Findeisen <pi...@gmail.com>
Co-authored-by: Simon Vandel Sillesen <si...@gmail.com>
Co-authored-by: aditya singh rathore <14...@users.noreply.github.com>
Co-authored-by: dennis zhuang <ki...@gmail.com>
Co-authored-by: Heran Lin <he...@lakesail.com>
Co-authored-by: Vyquos <75...@users.noreply.github.com>
Co-authored-by: Hiroaki Yutani <yu...@gmail.com>
Co-authored-by: LFC <99...@users.noreply.github.com>
Co-authored-by: Sai Mahendra <11...@users.noreply.github.com>
Co-authored-by: Renato Marroquin <re...@gmail.com>
Co-authored-by: Matt Butrovich <mb...@users.noreply.github.com>
Co-authored-by: Evgenii Glotov <gl...@gmail.com>
Co-authored-by: wiedld <wi...@users.noreply.github.com>
Co-authored-by: Devam Patel <11...@users.noreply.github.com>
Co-authored-by: Mason <mh...@gmail.com>
Co-authored-by: Berkay Şahin <12...@users.noreply.github.com>
Co-authored-by: Nihal Rajak <ni...@gmail.com>
Co-authored-by: Anders <an...@dbtlabs.com>
Co-authored-by: Shiv Bhatia <sh...@gmail.com>
Co-authored-by: Shiv Bhatia <sb...@palantir.com>
Co-authored-by: Ilya Ostanevich <86...@users.noreply.github.com>
Co-authored-by: Rok Mihevc <ro...@mihevc.org>
---
.github/actions/setup-macos-builder/action.yaml | 47 -
.github/actions/setup-rust-runtime/action.yaml | 9 -
.github/dependabot.yml | 2 +-
.github/workflows/audit.yml | 6 +-
.github/workflows/dev.yml | 5 +-
.github/workflows/rust.yml | 47 +-
.gitignore | 1 +
Cargo.lock | 1002 +++++++++++---------
Cargo.toml | 92 +-
benchmarks/Cargo.toml | 2 +-
benchmarks/README.md | 47 +
benchmarks/bench.sh | 61 +-
benchmarks/compile_profile.py | 180 ++++
benchmarks/src/bin/dfbench.rs | 6 +-
benchmarks/src/hj.rs | 273 ++++++
benchmarks/src/imdb/run.rs | 2 +-
benchmarks/src/lib.rs | 1 +
benchmarks/src/tpch/run.rs | 2 +-
.../scripts/license_header.sh | 11 +-
datafusion-cli/CONTRIBUTING.md | 10 +-
datafusion-cli/README.md | 7 +-
datafusion-cli/src/main.rs | 12 +-
.../examples/composed_extension_codec.rs | 14 +-
datafusion-examples/examples/custom_file_casts.rs | 10 +-
.../examples/date_time_functions.rs | 16 +-
datafusion-examples/examples/expr_api.rs | 2 +-
.../examples/flight/flight_server.rs | 2 +-
.../examples/flight/flight_sql_server.rs | 6 +-
datafusion/catalog-listing/Cargo.toml | 2 +-
datafusion/catalog-listing/README.md | 8 +-
datafusion/catalog/Cargo.toml | 2 +-
datafusion/catalog/README.md | 7 +-
datafusion/catalog/src/information_schema.rs | 5 +-
datafusion/catalog/src/listing_schema.rs | 1 +
datafusion/catalog/src/stream.rs | 2 +-
datafusion/catalog/src/view.rs | 2 +-
datafusion/common-runtime/README.md | 7 +-
datafusion/common/Cargo.toml | 2 +-
datafusion/common/README.md | 7 +-
datafusion/common/src/cast.rs | 19 +-
datafusion/common/src/config.rs | 8 +
datafusion/common/src/dfschema.rs | 99 +-
.../common/src/file_options/parquet_writer.rs | 13 +-
datafusion/common/src/nested_struct.rs | 12 +-
datafusion/common/src/param_value.rs | 2 +-
datafusion/common/src/scalar/mod.rs | 555 +++++++++--
datafusion/common/src/types/native.rs | 26 +-
datafusion/core/Cargo.toml | 3 +-
datafusion/core/README.md | 13 +-
datafusion/core/benches/sql_planner.rs | 6 +-
datafusion/core/src/dataframe/mod.rs | 36 +-
datafusion/core/src/dataframe/parquet.rs | 7 +-
datafusion/core/src/datasource/file_format/csv.rs | 2 +-
.../core/src/datasource/listing_table_factory.rs | 11 +-
.../core/src/datasource/physical_plan/parquet.rs | 4 +-
datafusion/core/src/execution/context/mod.rs | 58 +-
datafusion/core/src/execution/session_state.rs | 8 +
datafusion/core/src/physical_planner.rs | 272 +++---
datafusion/core/src/test_util/mod.rs | 2 +-
datafusion/core/tests/dataframe/mod.rs | 240 ++++-
.../tests/fuzz_cases/record_batch_generator.rs | 55 +-
.../core/tests/fuzz_cases/topk_filter_pushdown.rs | 4 +-
datafusion/core/tests/fuzz_cases/window_fuzz.rs | 6 +-
datafusion/core/tests/macro_hygiene/mod.rs | 1 +
.../memory_limit_validation/sort_mem_validation.rs | 8 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 101 +-
datafusion/core/tests/parquet/schema_adapter.rs | 4 +-
.../tests/physical_optimizer/enforce_sorting.rs | 2 +-
.../physical_optimizer/filter_pushdown/mod.rs | 34 +-
.../physical_optimizer/projection_pushdown.rs | 2 +-
.../core/tests/physical_optimizer/test_utils.rs | 4 +-
datafusion/core/tests/sql/sql_api.rs | 4 +-
.../tests/user_defined/user_defined_aggregates.rs | 14 +-
.../user_defined/user_defined_scalar_functions.rs | 95 +-
.../user_defined/user_defined_window_functions.rs | 32 +-
datafusion/datasource-avro/Cargo.toml | 2 +-
datafusion/datasource-avro/README.md | 10 +-
.../src/avro_to_arrow/arrow_array_reader.rs | 14 +-
datafusion/datasource-csv/Cargo.toml | 2 +-
datafusion/datasource-csv/README.md | 7 +-
datafusion/datasource-json/Cargo.toml | 2 +-
datafusion/datasource-json/README.md | 7 +-
datafusion/datasource-parquet/Cargo.toml | 2 +-
datafusion/datasource-parquet/README.md | 10 +-
datafusion/datasource-parquet/src/file_format.rs | 84 +-
datafusion/datasource-parquet/src/metadata.rs | 6 +-
datafusion/datasource-parquet/src/metrics.rs | 17 +
datafusion/datasource-parquet/src/opener.rs | 62 +-
datafusion/datasource-parquet/src/row_filter.rs | 5 +-
datafusion/datasource-parquet/src/source.rs | 7 +
datafusion/datasource/Cargo.toml | 2 +-
datafusion/datasource/README.md | 7 +-
datafusion/datasource/src/file_scan_config.rs | 61 +-
datafusion/datasource/src/schema_adapter.rs | 2 +-
datafusion/doc/Cargo.toml | 1 +
datafusion/doc/README.md | 7 +-
datafusion/doc/src/lib.rs | 8 +
.../src/function/map/mod.rs => doc/src/udaf.rs} | 32 +-
datafusion/doc/src/udf.rs | 132 +++
datafusion/doc/src/udwf.rs | 47 +
datafusion/execution/README.md | 7 +-
datafusion/execution/src/task.rs | 8 +
datafusion/expr-common/Cargo.toml | 1 +
datafusion/expr-common/README.md | 7 +-
datafusion/expr-common/src/casts.rs | 70 +-
datafusion/expr-common/src/signature.rs | 13 +-
datafusion/expr-common/src/statistics.rs | 2 +-
.../expr-common/src/type_coercion/aggregates.rs | 53 +-
datafusion/expr-common/src/type_coercion/binary.rs | 130 ++-
.../src/type_coercion/binary/tests/arithmetic.rs | 57 +-
datafusion/expr/Cargo.toml | 2 +-
datafusion/expr/README.md | 7 +-
datafusion/expr/src/conditional_expressions.rs | 8 +-
datafusion/expr/src/expr.rs | 4 +-
datafusion/expr/src/expr_schema.rs | 4 +-
datafusion/expr/src/lib.rs | 19 +-
datafusion/expr/src/logical_plan/builder.rs | 205 ++--
datafusion/expr/src/logical_plan/ddl.rs | 2 +
datafusion/expr/src/logical_plan/display.rs | 8 +-
datafusion/expr/src/logical_plan/plan.rs | 61 +-
datafusion/expr/src/logical_plan/statement.rs | 3 +-
datafusion/expr/src/registry.rs | 26 +-
datafusion/expr/src/test/function_stub.rs | 22 +-
datafusion/expr/src/type_coercion/functions.rs | 17 +-
datafusion/expr/src/type_coercion/mod.rs | 10 +-
datafusion/expr/src/udaf.rs | 48 +-
datafusion/expr/src/udf.rs | 116 ---
datafusion/expr/src/udwf.rs | 31 -
datafusion/expr/src/window_state.rs | 16 +-
datafusion/ffi/README.md | 22 +-
datafusion/ffi/src/plan_properties.rs | 5 +-
datafusion/ffi/src/udaf/accumulator_args.rs | 5 +-
datafusion/ffi/src/udwf/partition_evaluator.rs | 2 +-
.../ffi/src/udwf/partition_evaluator_args.rs | 2 +-
datafusion/functions-aggregate-common/Cargo.toml | 1 +
datafusion/functions-aggregate-common/README.md | 7 +-
.../src/aggregate/avg_distinct.rs | 2 +
.../src/aggregate/avg_distinct/decimal.rs | 282 ++++++
.../functions-aggregate-common/src/min_max.rs | 66 +-
.../functions-aggregate-common/src/tdigest.rs | 4 +-
datafusion/functions-aggregate/README.md | 7 +-
.../functions-aggregate/src/approx_distinct.rs | 60 +-
.../src/approx_percentile_cont.rs | 18 +-
.../src/approx_percentile_cont_with_weight.rs | 31 +-
datafusion/functions-aggregate/src/average.rs | 174 +++-
.../functions-aggregate/src/bit_and_or_xor.rs | 6 +-
datafusion/functions-aggregate/src/correlation.rs | 11 +-
datafusion/functions-aggregate/src/first_last.rs | 20 +-
datafusion/functions-aggregate/src/lib.rs | 2 +
datafusion/functions-aggregate/src/median.rs | 8 +-
datafusion/functions-aggregate/src/min_max.rs | 24 +-
datafusion/functions-aggregate/src/regr.rs | 2 +-
datafusion/functions-aggregate/src/sum.rs | 47 +-
datafusion/functions-nested/README.md | 10 +-
datafusion/functions-nested/src/array_has.rs | 74 +-
datafusion/functions-nested/src/cardinality.rs | 6 -
datafusion/functions-nested/src/concat.rs | 5 +-
datafusion/functions-nested/src/distance.rs | 6 +-
datafusion/functions-nested/src/except.rs | 8 +-
datafusion/functions-nested/src/extract.rs | 35 +-
datafusion/functions-nested/src/flatten.rs | 16 +-
datafusion/functions-nested/src/length.rs | 35 +-
datafusion/functions-nested/src/make_array.rs | 6 +-
datafusion/functions-nested/src/planner.rs | 2 +-
datafusion/functions-nested/src/position.rs | 4 +-
datafusion/functions-nested/src/remove.rs | 18 +-
datafusion/functions-nested/src/replace.rs | 6 +-
datafusion/functions-nested/src/resize.rs | 6 +-
datafusion/functions-nested/src/reverse.rs | 6 +-
datafusion/functions-nested/src/sort.rs | 63 +-
datafusion/functions-nested/src/string.rs | 104 +-
datafusion/functions-nested/src/utils.rs | 8 +-
datafusion/functions-table/README.md | 7 +-
datafusion/functions-window-common/README.md | 7 +-
datafusion/functions-window/README.md | 7 +-
datafusion/functions-window/src/lead_lag.rs | 2 +-
datafusion/functions-window/src/nth_value.rs | 2 +-
datafusion/functions-window/src/rank.rs | 2 +-
datafusion/functions/README.md | 7 +-
datafusion/functions/benches/regx.rs | 13 +-
datafusion/functions/src/core/getfield.rs | 4 +-
datafusion/functions/src/core/named_struct.rs | 2 +-
datafusion/functions/src/core/nvl2.rs | 2 +-
datafusion/functions/src/datetime/from_unixtime.rs | 4 +-
datafusion/functions/src/datetime/to_date.rs | 4 +-
datafusion/functions/src/datetime/to_timestamp.rs | 13 +-
datafusion/functions/src/datetime/to_unixtime.rs | 2 +-
datafusion/functions/src/macros.rs | 10 +-
datafusion/functions/src/math/abs.rs | 12 +-
datafusion/functions/src/math/cot.rs | 8 +
datafusion/functions/src/math/factorial.rs | 12 +-
datafusion/functions/src/math/gcd.rs | 8 +
datafusion/functions/src/math/iszero.rs | 8 +
datafusion/functions/src/math/lcm.rs | 12 +-
datafusion/functions/src/math/log.rs | 8 +
datafusion/functions/src/math/monotonicity.rs | 241 ++++-
datafusion/functions/src/math/nans.rs | 8 +
datafusion/functions/src/math/nanvl.rs | 8 +
datafusion/functions/src/math/power.rs | 12 +-
datafusion/functions/src/math/random.rs | 10 +-
datafusion/functions/src/math/round.rs | 10 +-
datafusion/functions/src/math/signum.rs | 10 +-
datafusion/functions/src/math/trunc.rs | 11 +-
datafusion/functions/src/regex/regexpreplace.rs | 136 ++-
datafusion/functions/src/string/concat.rs | 2 +-
datafusion/functions/src/string/contains.rs | 2 +-
datafusion/functions/src/string/replace.rs | 2 +-
datafusion/macros/Cargo.toml | 5 +-
datafusion/macros/README.md | 11 +-
datafusion/macros/src/user_doc.rs | 2 +-
datafusion/optimizer/README.md | 7 +-
datafusion/optimizer/src/analyzer/type_coercion.rs | 47 +-
.../optimizer/src/common_subexpr_eliminate.rs | 26 +-
.../optimizer/src/extract_equijoin_predicate.rs | 123 ++-
datafusion/optimizer/src/optimizer.rs | 7 +-
datafusion/optimizer/src/push_down_filter.rs | 71 +-
.../src/simplify_expressions/expr_simplifier.rs | 228 ++++-
.../src/simplify_expressions/unwrap_cast.rs | 2 +-
.../optimizer/src/simplify_expressions/utils.rs | 27 +-
datafusion/physical-expr-adapter/README.md | 32 +-
datafusion/physical-expr-common/README.md | 11 +-
datafusion/physical-expr/README.md | 7 +-
datafusion/physical-expr/src/expressions/binary.rs | 10 +-
.../src/expressions/binary/kernels.rs | 4 +-
datafusion/physical-expr/src/expressions/cast.rs | 2 +-
.../src/expressions/dynamic_filters.rs | 39 +-
.../physical-expr/src/expressions/in_list.rs | 4 +-
.../physical-expr/src/expressions/try_cast.rs | 2 +-
datafusion/physical-expr/src/planner.rs | 3 +-
datafusion/physical-expr/src/utils/mod.rs | 33 +-
datafusion/physical-expr/src/window/standard.rs | 8 +-
datafusion/physical-optimizer/README.md | 8 +-
datafusion/physical-optimizer/src/lib.rs | 2 +-
datafusion/physical-plan/README.md | 7 +-
datafusion/physical-plan/benches/spill_io.rs | 14 +-
datafusion/physical-plan/src/display.rs | 2 +-
datafusion/physical-plan/src/filter.rs | 14 +-
datafusion/physical-plan/src/filter_pushdown.rs | 4 +-
.../physical-plan/src/joins/hash_join/exec.rs | 19 +-
.../physical-plan/src/joins/nested_loop_join.rs | 25 +-
.../src/joins/sort_merge_join/exec.rs | 25 +-
datafusion/physical-plan/src/joins/utils.rs | 35 +-
datafusion/physical-plan/src/limit.rs | 3 +-
.../physical-plan/src/sorts/multi_level_merge.rs | 9 +-
datafusion/physical-plan/src/spill/mod.rs | 49 +-
.../physical-plan/src/spill/spill_manager.rs | 2 +
.../src/windows/bounded_window_agg_exec.rs | 51 +-
datafusion/physical-plan/src/windows/mod.rs | 158 ++-
datafusion/proto-common/Cargo.toml | 2 +-
datafusion/proto-common/README.md | 14 +-
datafusion/proto-common/gen/Cargo.toml | 4 +-
.../proto-common/proto/datafusion_common.proto | 4 +
datafusion/proto-common/src/from_proto/mod.rs | 16 +-
datafusion/proto-common/src/generated/pbjson.rs | 26 +-
datafusion/proto-common/src/generated/prost.rs | 111 ++-
datafusion/proto-common/src/to_proto/mod.rs | 39 +-
datafusion/proto/Cargo.toml | 9 +-
datafusion/proto/README.md | 14 +-
datafusion/proto/gen/Cargo.toml | 4 +-
datafusion/proto/proto/datafusion.proto | 1 +
datafusion/proto/src/bytes/mod.rs | 17 +-
datafusion/proto/src/bytes/registry.rs | 8 +
.../proto/src/generated/datafusion_proto_common.rs | 111 ++-
datafusion/proto/src/generated/pbjson.rs | 18 +
datafusion/proto/src/generated/prost.rs | 78 +-
datafusion/proto/src/lib.rs | 2 +-
datafusion/proto/src/logical_plan/file_formats.rs | 283 +++---
datafusion/proto/src/logical_plan/mod.rs | 3 +
datafusion/proto/src/physical_plan/from_proto.rs | 31 +-
datafusion/proto/src/physical_plan/mod.rs | 400 ++++----
.../proto/tests/cases/roundtrip_logical_plan.rs | 9 +-
.../proto/tests/cases/roundtrip_physical_plan.rs | 42 +-
datafusion/pruning/Cargo.toml | 1 +
datafusion/{catalog => pruning}/README.md | 11 +-
datafusion/pruning/src/pruning_predicate.rs | 8 +-
datafusion/session/Cargo.toml | 2 +-
datafusion/session/README.md | 7 +-
datafusion/spark/README.md | 19 +-
datafusion/spark/src/function/array/spark_array.rs | 2 +-
.../spark/src/function/bitmap/bitmap_count.rs | 26 +-
datafusion/spark/src/function/bitwise/bit_count.rs | 2 +-
datafusion/spark/src/function/bitwise/bit_shift.rs | 2 +-
datafusion/spark/src/function/datetime/date_sub.rs | 2 +-
.../src/function/datetime/make_dt_interval.rs | 485 ++++++++++
.../spark/src/function/datetime/make_interval.rs | 581 ++++++++++++
datafusion/spark/src/function/datetime/mod.rs | 23 +-
.../spark/src/function/map/map_from_arrays.rs | 105 ++
.../spark/src/function/map/map_from_entries.rs | 133 +++
datafusion/spark/src/function/map/mod.rs | 26 +-
datafusion/spark/src/function/map/utils.rs | 231 +++++
datafusion/spark/src/function/math/factorial.rs | 4 +-
datafusion/spark/src/function/math/hex.rs | 7 +-
datafusion/spark/src/function/string/luhn_check.rs | 2 +-
datafusion/spark/src/lib.rs | 2 +
datafusion/sql/Cargo.toml | 4 +-
datafusion/sql/README.md | 15 +-
datafusion/sql/src/cte.rs | 8 +-
datafusion/sql/src/expr/function.rs | 63 +-
datafusion/sql/src/parser.rs | 145 ++-
datafusion/sql/src/planner.rs | 4 +-
datafusion/sql/src/query.rs | 78 +-
datafusion/sql/src/relation/mod.rs | 2 +-
datafusion/sql/src/select.rs | 10 +-
datafusion/sql/src/statement.rs | 3 +-
datafusion/sql/src/unparser/dialect.rs | 18 +
datafusion/sql/src/unparser/expr.rs | 184 +++-
datafusion/sql/tests/cases/params.rs | 23 +-
datafusion/sql/tests/sql_integration.rs | 2 +-
datafusion/sqllogictest/Cargo.toml | 6 +-
datafusion/sqllogictest/README.md | 22 +-
datafusion/sqllogictest/src/test_context.rs | 61 +-
datafusion/sqllogictest/test_files/aggregate.slt | 257 +++--
.../test_files/aggregate_skip_partial.slt | 7 -
datafusion/sqllogictest/test_files/array.slt | 268 +++++-
datafusion/sqllogictest/test_files/async_udf.slt | 107 +++
.../test_files/create_external_table.slt | 6 +
.../sqllogictest/test_files/create_function.slt | 5 -
datafusion/sqllogictest/test_files/ddl.slt | 53 +-
datafusion/sqllogictest/test_files/errors.slt | 2 +-
.../sqllogictest/test_files/explain_tree.slt | 14 +-
datafusion/sqllogictest/test_files/expr.slt | 6 -
datafusion/sqllogictest/test_files/group_by.slt | 8 -
.../sqllogictest/test_files/information_schema.slt | 2 +
datafusion/sqllogictest/test_files/join.slt.part | 10 +
.../test_files/join_is_not_distinct_from.slt | 321 +++++++
datafusion/sqllogictest/test_files/limit.slt | 2 +-
.../sqllogictest/test_files/pipe_operator.slt | 91 ++
.../sqllogictest/test_files/push_down_filter.slt | 2 +-
.../test_files/spark/datetime/make_dt_interval.slt | 125 ++-
.../test_files/spark/datetime/make_interval.slt | 112 +++
.../test_files/spark/map/map_from_arrays.slt | 136 +++
.../test_files/spark/map/map_from_entries.slt | 164 ++++
.../test_files/string/string_literal.slt | 20 +
.../sqllogictest/test_files/string/string_view.slt | 2 +-
datafusion/sqllogictest/test_files/timestamps.slt | 33 +
datafusion/sqllogictest/test_files/topk.slt | 16 +-
datafusion/sqllogictest/test_files/union.slt | 8 +-
datafusion/sqllogictest/test_files/window.slt | 89 ++
datafusion/substrait/README.md | 7 +-
.../src/logical_plan/consumer/expr/literal.rs | 42 +-
.../src/logical_plan/consumer/expr/mod.rs | 2 +-
.../src/logical_plan/consumer/expr/subquery.rs | 5 +-
.../src/logical_plan/consumer/rel/join_rel.rs | 4 +-
.../src/logical_plan/consumer/rel/read_rel.rs | 2 +-
.../src/logical_plan/producer/expr/literal.rs | 45 +-
.../src/logical_plan/producer/rel/aggregate_rel.rs | 2 +-
.../src/logical_plan/producer/rel/join.rs | 5 +-
.../substrait/src/logical_plan/producer/types.rs | 4 +-
datafusion/substrait/src/physical_plan/consumer.rs | 2 +-
.../substrait/tests/cases/consumer_integration.rs | 42 +-
.../tests/cases/roundtrip_logical_plan.rs | 90 +-
.../substrait/tests/cases/substrait_validations.rs | 2 +-
datafusion/substrait/tests/utils.rs | 2 +-
datafusion/wasmtest/Cargo.toml | 2 +-
datafusion/wasmtest/README.md | 16 +-
dev/release/README.md | 43 +-
dev/release/publish_homebrew.sh | 92 --
dev/rust_lint.sh | 8 +
docs/scripts/update_committer_list.py | 266 ++++++
docs/source/_static/theme_overrides.css | 26 +
docs/source/contributor-guide/governance.md | 82 +-
.../gsoc_application_guidelines_2025.md} | 2 +-
.../gsoc_project_ideas_2025.md} | 2 +-
docs/source/contributor-guide/gsoc/index.rst | 36 +
docs/source/index.rst | 7 +-
docs/source/library-user-guide/query-optimizer.md | 2 +-
docs/source/library-user-guide/upgrading.md | 51 +-
docs/source/user-guide/configs.md | 1 +
docs/source/user-guide/expressions.md | 4 +-
docs/source/user-guide/introduction.md | 48 +-
docs/source/user-guide/sql/scalar_functions.md | 396 ++++++++
docs/source/user-guide/sql/select.md | 96 ++
rust-toolchain.toml | 2 +-
test-utils/src/array_gen/random_data.rs | 14 +-
testing | 2 +-
typos.toml | 3 +-
376 files changed, 11779 insertions(+), 3266 deletions(-)
diff --git a/.github/actions/setup-macos-builder/action.yaml b/.github/actions/setup-macos-builder/action.yaml
deleted file mode 100644
index fffdab160..000000000
--- a/.github/actions/setup-macos-builder/action.yaml
+++ /dev/null
@@ -1,47 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-name: Prepare Rust Builder for MacOS
-description: 'Prepare Rust Build Environment for MacOS'
-inputs:
- rust-version:
- description: 'version of rust to install (e.g. stable)'
- required: true
- default: 'stable'
-runs:
- using: "composite"
- steps:
- - name: Install protobuf compiler
- shell: bash
- run: |
- mkdir -p $HOME/d/protoc
- cd $HOME/d/protoc
- export PROTO_ZIP="protoc-29.1-osx-x86_64.zip"
- curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v29.1/$PROTO_ZIP
- unzip $PROTO_ZIP
- echo "$HOME/d/protoc/bin" >> $GITHUB_PATH
- export PATH=$PATH:$HOME/d/protoc/bin
- protoc --version
- - name: Setup Rust toolchain
- shell: bash
- run: |
- rustup update stable
- rustup toolchain install stable
- rustup default stable
- rustup component add rustfmt
- - name: Configure rust runtime env
- uses: ./.github/actions/setup-rust-runtime
diff --git a/.github/actions/setup-rust-runtime/action.yaml b/.github/actions/setup-rust-runtime/action.yaml
index b6fb2c898..e0341de93 100644
--- a/.github/actions/setup-rust-runtime/action.yaml
+++ b/.github/actions/setup-rust-runtime/action.yaml
@@ -20,10 +20,6 @@ description: 'Setup Rust Runtime Environment'
runs:
using: "composite"
steps:
- # https://github.com/apache/datafusion/issues/15535
- # disabled because neither version nor git hash works with apache github policy
- #- name: Run sccache-cache
- # uses: mozilla-actions/sccache-action@65101d47ea8028ed0c98a1cdea8dd9182e9b5133 # v0.0.8
- name: Configure runtime env
shell: bash
# do not produce debug symbols to keep memory usage down
@@ -32,11 +28,6 @@ runs:
#
# Set debuginfo=line-tables-only as debuginfo=0 causes immensely slow build
# See for more details: https://github.com/rust-lang/rust/issues/119560
- #
- # readd the following to the run below once sccache-cache is re-enabled
- # echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV
- # echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
run: |
echo "RUST_BACKTRACE=1" >> $GITHUB_ENV
echo "RUSTFLAGS=-C debuginfo=line-tables-only -C incremental=false" >> $GITHUB_ENV
-
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 7c2b7e3a5..ec3368da9 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -20,7 +20,7 @@ updates:
- package-ecosystem: cargo
directory: "/"
schedule:
- interval: daily
+ interval: weekly
target-branch: main
labels: [auto-dependencies]
ignore:
diff --git a/.github/workflows/audit.yml b/.github/workflows/audit.yml
index b26ac8c02..d05141c2e 100644
--- a/.github/workflows/audit.yml
+++ b/.github/workflows/audit.yml
@@ -42,10 +42,8 @@ jobs:
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Install cargo-audit
- uses: taiki-e/install-action@2fdc5fd6ac805b0f8256893bd4c807bcb666af00 # v2
+ uses: taiki-e/install-action@d0f4f69b07c0804d1003ca9a5a5f853423872ed9 # v2
with:
tool: cargo-audit
- name: Run audit check
- # Ignored until https://github.com/apache/datafusion/issues/15571
- # ignored py03 warning until arrow 55 upgrade
- run: cargo audit --ignore RUSTSEC-2024-0370 --ignore RUSTSEC-2025-0020
+ run: cargo audit
diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml
index 97f1b4e5b..103577290 100644
--- a/.github/workflows/dev.yml
+++ b/.github/workflows/dev.yml
@@ -33,7 +33,10 @@ jobs:
name: Check License Header
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- - uses: korandoru/hawkeye@c3d9887d4b48e5b6c77306934606a4ff4623a2d3 # v6.2.0
+ - name: Install HawkEye
+ run: cargo install hawkeye --version 6.2.0 --locked --profile dev
+ - name: Run license header check
+ run: ci/scripts/license_header.sh
prettier:
name: Use prettier to check formatting of documents
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 30fd684a0..154843944 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -55,7 +55,7 @@ jobs:
with:
rust-version: stable
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
shared-key: "amd-ci-check" # this job uses it's own cache becase check has a separate cache and we need it to be fast as it blocks other jobs
save-if: ${{ github.ref_name == 'main' }}
@@ -108,7 +108,7 @@ jobs:
with:
rust-version: stable
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: false # set in linux-test
shared-key: "amd-ci"
@@ -176,7 +176,7 @@ jobs:
with:
rust-version: stable
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: false # set in linux-test
shared-key: "amd-ci"
@@ -281,7 +281,7 @@ jobs:
with:
rust-version: stable
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci"
@@ -308,10 +308,6 @@ jobs:
name: cargo test datafusion-cli (amd64)
needs: linux-build-lib
runs-on: ubuntu-latest
- # should be uncommented once https://github.com/apache/datafusion/pull/16644 is merged
- # and cache should be added
- # container:
- # image: amd64/rust
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
@@ -319,6 +315,11 @@ jobs:
fetch-depth: 1
- name: Setup Rust toolchain
run: rustup toolchain install stable
+ - name: Rust Dependency Cache
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
+ with:
+ save-if: false # set in linux-test
+ shared-key: "amd-ci"
- name: Run tests (excluding doctests)
env:
RUST_BACKTRACE: 1
@@ -348,7 +349,7 @@ jobs:
with:
rust-version: stable
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci-linux-test-example"
@@ -411,14 +412,16 @@ jobs:
sudo apt-get update -qq
sudo apt-get install -y -qq clang
- name: Setup wasm-pack
- uses: taiki-e/install-action@2fdc5fd6ac805b0f8256893bd4c807bcb666af00 # v2
+ uses: taiki-e/install-action@d0f4f69b07c0804d1003ca9a5a5f853423872ed9 # v2
with:
tool: wasm-pack
- name: Run tests with headless mode
working-directory: ./datafusion/wasmtest
run: |
- RUSTFLAGS='--cfg getrandom_backend="wasm_js"' wasm-pack test --headless --firefox
- RUSTFLAGS='--cfg getrandom_backend="wasm_js"' wasm-pack test --headless --chrome --chromedriver $CHROMEWEBDRIVER/chromedriver
+ # debuginfo=none because CI tests weren't completing successfully after this upstream PR:
+ # https://github.com/wasm-bindgen/wasm-bindgen/pull/4635
+ RUSTFLAGS='--cfg getrandom_backend="wasm_js" -C debuginfo=none' wasm-pack test --headless --firefox
+ RUSTFLAGS='--cfg getrandom_backend="wasm_js" -C debuginfo=none' wasm-pack test --headless --chrome --chromedriver $CHROMEWEBDRIVER/chromedriver
# verify that the benchmark queries return the correct results
verify-benchmark-results:
@@ -533,22 +536,6 @@ jobs:
# export PATH=$PATH:$HOME/d/protoc/bin
# cargo test --lib --tests --bins --features avro,json,backtrace
- # Commenting out intel mac build as so few users would ever use it
- # Details: https://github.com/apache/datafusion/issues/13846
- # macos:
- # name: cargo test (macos)
- # runs-on: macos-latest
- # steps:
- # - uses: actions/checkout@v4
- # with:
- # submodules: true
- # fetch-depth: 1
- # - name: Setup Rust toolchain
- # uses: ./.github/actions/setup-macos-builder
- # - name: Run tests (excluding doctests)
- # shell: bash
- # run: cargo test run --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace
-
macos-aarch64:
name: cargo test (macos-aarch64)
runs-on: macos-14
@@ -680,7 +667,7 @@ jobs:
- name: Install Clippy
run: rustup component add clippy
- name: Rust Dependency Cache
- uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
+ uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci-clippy"
@@ -752,7 +739,7 @@ jobs:
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
- name: Install cargo-msrv
- uses: taiki-e/install-action@2fdc5fd6ac805b0f8256893bd4c807bcb666af00 # v2
+ uses: taiki-e/install-action@d0f4f69b07c0804d1003ca9a5a5f853423872ed9 # v2
with:
tool: cargo-msrv
diff --git a/.gitignore b/.gitignore
index 4ae32925d..8466a72ad 100644
--- a/.gitignore
+++ b/.gitignore
@@ -31,6 +31,7 @@ docker_cache
*.orig
.*.swp
.*.swo
+*.pending-snap
venv/*
diff --git a/Cargo.lock b/Cargo.lock
index 9dd40437f..fe0f93e1f 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -187,9 +187,9 @@ dependencies = [
[[package]]
name = "anyhow"
-version = "1.0.99"
+version = "1.0.100"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0674a1ddeecb70197781e945de4b3b8ffb61fa939a5597bcf48503737663100"
+checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
[[package]]
name = "apache-avro"
@@ -212,8 +212,8 @@ dependencies = [
"serde_bytes",
"serde_json",
"snap",
- "strum",
- "strum_macros",
+ "strum 0.27.2",
+ "strum_macros 0.27.2",
"thiserror",
"uuid",
"xz2",
@@ -234,9 +234,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "arrow"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd798aea3553913a5986813e9c6ad31a2d2b04e931fe8ea4a37155eb541cebb5"
+checksum = "6e833808ff2d94ed40d9379848a950d995043c7fb3e81a30b383f4c6033821cc"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -258,9 +258,9 @@ dependencies = [
[[package]]
name = "arrow-arith"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "508dafb53e5804a238cab7fd97a59ddcbfab20cc4d9814b1ab5465b9fa147f2e"
+checksum = "ad08897b81588f60ba983e3ca39bda2b179bdd84dced378e7df81a5313802ef8"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -272,9 +272,9 @@ dependencies = [
[[package]]
name = "arrow-array"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e2730bc045d62bb2e53ef8395b7d4242f5c8102f41ceac15e8395b9ac3d08461"
+checksum = "8548ca7c070d8db9ce7aa43f37393e4bfcf3f2d3681df278490772fd1673d08d"
dependencies = [
"ahash 0.8.12",
"arrow-buffer",
@@ -283,15 +283,15 @@ dependencies = [
"chrono",
"chrono-tz",
"half",
- "hashbrown 0.15.5",
+ "hashbrown 0.16.0",
"num",
]
[[package]]
name = "arrow-buffer"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "54295b93beb702ee9a6f6fbced08ad7f4d76ec1c297952d4b83cf68755421d1d"
+checksum = "e003216336f70446457e280807a73899dd822feaf02087d31febca1363e2fccc"
dependencies = [
"bytes",
"half",
@@ -300,9 +300,9 @@ dependencies = [
[[package]]
name = "arrow-cast"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "67e8bcb7dc971d779a7280593a1bf0c2743533b8028909073e804552e85e75b5"
+checksum = "919418a0681298d3a77d1a315f625916cb5678ad0d74b9c60108eb15fd083023"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -321,9 +321,9 @@ dependencies = [
[[package]]
name = "arrow-csv"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "673fd2b5fb57a1754fdbfac425efd7cf54c947ac9950c1cce86b14e248f1c458"
+checksum = "bfa9bf02705b5cf762b6f764c65f04ae9082c7cfc4e96e0c33548ee3f67012eb"
dependencies = [
"arrow-array",
"arrow-cast",
@@ -336,9 +336,9 @@ dependencies = [
[[package]]
name = "arrow-data"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "97c22fe3da840039c69e9f61f81e78092ea36d57037b4900151f063615a2f6b4"
+checksum = "a5c64fff1d142f833d78897a772f2e5b55b36cb3e6320376f0961ab0db7bd6d0"
dependencies = [
"arrow-buffer",
"arrow-schema",
@@ -348,9 +348,9 @@ dependencies = [
[[package]]
name = "arrow-flight"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6808d235786b721e49e228c44dd94242f2e8b46b7e95b233b0733c46e758bfee"
+checksum = "8c8b0ba0784d56bc6266b79f5de7a24b47024e7b3a0045d2ad4df3d9b686099f"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -368,21 +368,22 @@ dependencies = [
"futures",
"once_cell",
"paste",
- "prost",
- "prost-types",
+ "prost 0.13.5",
+ "prost-types 0.13.5",
"tonic",
]
[[package]]
name = "arrow-ipc"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "778de14c5a69aedb27359e3dd06dd5f9c481d5f6ee9fbae912dba332fd64636b"
+checksum = "1d3594dcddccc7f20fd069bc8e9828ce37220372680ff638c5e00dea427d88f5"
dependencies = [
"arrow-array",
"arrow-buffer",
"arrow-data",
"arrow-schema",
+ "arrow-select",
"flatbuffers",
"lz4_flex",
"zstd",
@@ -390,9 +391,9 @@ dependencies = [
[[package]]
name = "arrow-json"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3860db334fe7b19fcf81f6b56f8d9d95053f3839ffe443d56b5436f7a29a1794"
+checksum = "88cf36502b64a127dc659e3b305f1d993a544eab0d48cce704424e62074dc04b"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -401,7 +402,7 @@ dependencies = [
"arrow-schema",
"chrono",
"half",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"lexical-core",
"memchr",
"num",
@@ -412,9 +413,9 @@ dependencies = [
[[package]]
name = "arrow-ord"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "425fa0b42a39d3ff55160832e7c25553e7f012c3f187def3d70313e7a29ba5d9"
+checksum = "3c8f82583eb4f8d84d4ee55fd1cb306720cddead7596edce95b50ee418edf66f"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -425,9 +426,9 @@ dependencies = [
[[package]]
name = "arrow-pyarrow"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d944d8ae9b77230124e6570865b570416c33a5809f32c4136c679bbe774e45c9"
+checksum = "7d924b32e96f8bb74d94cd82bd97b313c432fcb0ea331689ef9e7c6b8be4b258"
dependencies = [
"arrow-array",
"arrow-data",
@@ -437,9 +438,9 @@ dependencies = [
[[package]]
name = "arrow-row"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "df9c9423c9e71abd1b08a7f788fcd203ba2698ac8e72a1f236f1faa1a06a7414"
+checksum = "9d07ba24522229d9085031df6b94605e0f4b26e099fb7cdeec37abd941a73753"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -450,20 +451,20 @@ dependencies = [
[[package]]
name = "arrow-schema"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "85fa1babc4a45fdc64a92175ef51ff00eba5ebbc0007962fecf8022ac1c6ce28"
+checksum = "b3aa9e59c611ebc291c28582077ef25c97f1975383f1479b12f3b9ffee2ffabe"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"serde",
"serde_json",
]
[[package]]
name = "arrow-select"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d8854d15f1cf5005b4b358abeb60adea17091ff5bdd094dca5d3f73787d81170"
+checksum = "8c41dbbd1e97bfcaee4fcb30e29105fb2c75e4d82ae4de70b792a5d3f66b2e7a"
dependencies = [
"ahash 0.8.12",
"arrow-array",
@@ -475,9 +476,9 @@ dependencies = [
[[package]]
name = "arrow-string"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c477e8b89e1213d5927a2a84a72c384a9bf4dd0dbf15f9fd66d821aafd9e95e"
+checksum = "53f5183c150fbc619eede22b861ea7c0eebed8eaac0333eaa7f6da5205fd504d"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -615,9 +616,9 @@ dependencies = [
[[package]]
name = "aws-lc-rs"
-version = "1.13.3"
+version = "1.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5c953fe1ba023e6b7730c0d4b031d06f267f23a46167dcbd40316644b10a17ba"
+checksum = "94b8ff6c09cd57b16da53641caa860168b88c172a5ee163b0288d3d6eea12786"
dependencies = [
"aws-lc-sys",
"zeroize",
@@ -625,9 +626,9 @@ dependencies = [
[[package]]
name = "aws-lc-sys"
-version = "0.30.0"
+version = "0.31.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dbfd150b5dbdb988bcc8fb1fe787eb6b7ee6180ca24da683b61ea5405f3d43ff"
+checksum = "0e44d16778acaf6a9ec9899b92cebd65580b83f685446bf2e1f5d3d732f99dcd"
dependencies = [
"bindgen",
"cc",
@@ -662,9 +663,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
-version = "1.83.0"
+version = "1.84.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "643cd43af212d2a1c4dedff6f044d7e1961e5d9e7cfe773d70f31d9842413886"
+checksum = "357a841807f6b52cb26123878b3326921e2a25faca412fabdd32bd35b7edd5d3"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -684,9 +685,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
-version = "1.84.0"
+version = "1.85.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20ec4a95bd48e0db7a424356a161f8d87bd6a4f0af37204775f0da03d9e39fc3"
+checksum = "67e05f33b6c9026fecfe9b3b6740f34d41bc6ff641a6a32dabaab60209245b75"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -706,9 +707,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
-version = "1.85.0"
+version = "1.86.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "410309ad0df4606bc721aff0d89c3407682845453247213a0ccc5ff8801ee107"
+checksum = "e7d835f123f307cafffca7b9027c14979f1d403b417d8541d67cf252e8a21e35"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -834,9 +835,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime"
-version = "1.9.1"
+version = "1.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d3946acbe1ead1301ba6862e712c7903ca9bb230bdf1fbd1b5ac54158ef2ab1f"
+checksum = "4fa63ad37685ceb7762fa4d73d06f1d5493feb88e3f27259b9ed277f4c01b185"
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
@@ -1017,25 +1018,22 @@ dependencies = [
[[package]]
name = "bindgen"
-version = "0.69.5"
+version = "0.72.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088"
+checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"cexpr",
"clang-sys",
- "itertools 0.12.1",
- "lazy_static",
- "lazycell",
+ "itertools 0.13.0",
"log",
"prettyplease",
"proc-macro2",
"quote",
"regex",
- "rustc-hash 1.1.0",
+ "rustc-hash",
"shlex",
"syn 2.0.106",
- "which",
]
[[package]]
@@ -1046,9 +1044,9 @@ checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
[[package]]
name = "bitflags"
-version = "2.9.1"
+version = "2.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1b8e56985ec62d17e9c1001dc89c88ecd7dc08e47eba5ec7c29c7b5eeecde967"
+checksum = "2261d10cca569e4643e526d8dc2e62e433cc8aba21ab764233731f8d369bf394"
[[package]]
name = "bitvec"
@@ -1145,9 +1143,9 @@ dependencies = [
[[package]]
name = "bon"
-version = "3.7.1"
+version = "3.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "537c317ddf588aab15c695bf92cf55dec159b93221c074180ca3e0e5a94da415"
+checksum = "c2529c31017402be841eb45892278a6c21a000c0a17643af326c73a73f83f0fb"
dependencies = [
"bon-macros",
"rustversion",
@@ -1155,11 +1153,11 @@ dependencies = [
[[package]]
name = "bon-macros"
-version = "3.7.1"
+version = "3.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ca5abbf2d4a4c6896197c9de13d6d7cb7eff438c63dacde1dde980569cb00248"
+checksum = "d82020dadcb845a345591863adb65d74fa8dc5c18a0b6d408470e13b7adc7005"
dependencies = [
- "darling 0.21.3",
+ "darling",
"ident_case",
"prettyplease",
"proc-macro2",
@@ -1193,9 +1191,9 @@ dependencies = [
[[package]]
name = "brotli"
-version = "8.0.1"
+version = "8.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9991eea70ea4f293524138648e41ee89b0b2b12ddef3b255effa43c8056e0e0d"
+checksum = "4bd8b9603c7aa97359dbd97ecf258968c95f3adddd6db2f7e7a5bef101c84560"
dependencies = [
"alloc-no-stdlib",
"alloc-stdlib",
@@ -1308,10 +1306,11 @@ checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
[[package]]
name = "cc"
-version = "1.2.32"
+version = "1.2.38"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2352e5597e9c544d5e6d9c95190d5d27738ade584fa8db0a16e130e5c2b5296e"
+checksum = "80f41ae168f955c12fb8960b057d70d0ca153fb83182b57d86380443527be7e9"
dependencies = [
+ "find-msvc-tools",
"jobserver",
"libc",
"shlex",
@@ -1328,9 +1327,9 @@ dependencies = [
[[package]]
name = "cfg-if"
-version = "1.0.1"
+version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9555578bc9e57714c812a1f84e4fc5b4d21fcb063490c624de019f7464c91268"
+checksum = "2fd1289c04a9ea8cb22300a459a72a385d7c73d3259e2ed7dcb2af674838cfa9"
[[package]]
name = "cfg_aliases"
@@ -1397,7 +1396,7 @@ checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
dependencies = [
"glob",
"libc",
- "libloading 0.8.8",
+ "libloading 0.8.9",
]
[[package]]
@@ -1413,9 +1412,9 @@ dependencies = [
[[package]]
name = "clap"
-version = "4.5.47"
+version = "4.5.48"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7eac00902d9d136acd712710d71823fb8ac8004ca445a89e73a41d45aa712931"
+checksum = "e2134bb3ea021b78629caa971416385309e0131b351b25e01dc16fb54e1b5fae"
dependencies = [
"clap_builder",
"clap_derive",
@@ -1423,9 +1422,9 @@ dependencies = [
[[package]]
name = "clap_builder"
-version = "4.5.47"
+version = "4.5.48"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2ad9bbf750e73b5884fb8a211a9424a1906c1e156724260fdae972f31d70e1d6"
+checksum = "c2ba64afa3c0a6df7fa517765e31314e983f51dda798ffba27b988194fb65dc9"
dependencies = [
"anstream",
"anstyle",
@@ -1477,11 +1476,12 @@ checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
[[package]]
name = "comfy-table"
-version = "7.1.4"
+version = "7.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4a65ebfec4fb190b6f90e944a817d60499ee0744e582530e2c9900a22e591d9a"
+checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56"
dependencies = [
- "unicode-segmentation",
+ "strum 0.26.3",
+ "strum_macros 0.26.4",
"unicode-width 0.2.1",
]
@@ -1499,15 +1499,15 @@ dependencies = [
[[package]]
name = "console"
-version = "0.16.0"
+version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e09ced7ebbccb63b4c65413d821f2e00ce54c5ca4514ddc6b3c892fdbcbc69d"
+checksum = "b430743a6eb14e9764d4260d4c0d8123087d504eeb9c48f2b2a5e810dd369df4"
dependencies = [
"encode_unicode",
"libc",
"once_cell",
"unicode-width 0.2.1",
- "windows-sys 0.60.2",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -1542,9 +1542,9 @@ dependencies = [
[[package]]
name = "const_panic"
-version = "0.2.14"
+version = "0.2.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bb8a602185c3c95b52f86dc78e55a6df9a287a7a93ddbcf012509930880cf879"
+checksum = "e262cdaac42494e3ae34c43969f9cdeb7da178bdb4b66fa6a1ea2edb4c8ae652"
dependencies = [
"typewit",
]
@@ -1573,18 +1573,18 @@ checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "core_extensions"
-version = "1.5.3"
+version = "1.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "92c71dc07c9721607e7a16108336048ee978c3a8b129294534272e8bac96c0ee"
+checksum = "42bb5e5d0269fd4f739ea6cedaf29c16d81c27a7ce7582008e90eb50dcd57003"
dependencies = [
"core_extensions_proc_macros",
]
[[package]]
name = "core_extensions_proc_macros"
-version = "1.5.3"
+version = "1.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "69f3b219d28b6e3b4ac87bc1fc522e0803ab22e055da177bff0068c4150c61a6"
+checksum = "533d38ecd2709b7608fb8e18e4504deb99e9a72879e6aa66373a76d8dc4259ea"
[[package]]
name = "cpufeatures"
@@ -1613,7 +1613,7 @@ dependencies = [
"anes",
"cast",
"ciborium",
- "clap 4.5.47",
+ "clap 4.5.48",
"criterion-plot",
"futures",
"is-terminal",
@@ -1735,38 +1735,14 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b365fabc795046672053e29c954733ec3b05e4be654ab130fe8f1f94d7051f35"
-[[package]]
-name = "darling"
-version = "0.20.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fc7f46116c46ff9ab3eb1597a45688b6715c6e628b5c133e288e709a29bcb4ee"
-dependencies = [
- "darling_core 0.20.11",
- "darling_macro 0.20.11",
-]
-
[[package]]
name = "darling"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
dependencies = [
- "darling_core 0.21.3",
- "darling_macro 0.21.3",
-]
-
-[[package]]
-name = "darling_core"
-version = "0.20.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d00b9596d185e565c2207a0b01f8bd1a135483d02d9b7b0a54b11da8d53412e"
-dependencies = [
- "fnv",
- "ident_case",
- "proc-macro2",
- "quote",
- "strsim",
- "syn 2.0.106",
+ "darling_core",
+ "darling_macro",
]
[[package]]
@@ -1783,24 +1759,13 @@ dependencies = [
"syn 2.0.106",
]
-[[package]]
-name = "darling_macro"
-version = "0.20.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead"
-dependencies = [
- "darling_core 0.20.11",
- "quote",
- "syn 2.0.106",
-]
-
[[package]]
name = "darling_macro"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
dependencies = [
- "darling_core 0.21.3",
+ "darling_core",
"quote",
"syn 2.0.106",
]
@@ -1968,7 +1933,7 @@ dependencies = [
"async-trait",
"aws-config",
"aws-credential-types",
- "clap 4.5.47",
+ "clap 4.5.48",
"ctor",
"datafusion",
"dirs",
@@ -2002,7 +1967,7 @@ dependencies = [
"half",
"hashbrown 0.14.5",
"hex",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
"libc",
"log",
@@ -2175,7 +2140,7 @@ dependencies = [
"mimalloc",
"nix",
"object_store",
- "prost",
+ "prost 0.13.5",
"rand 0.9.2",
"serde_json",
"tempfile",
@@ -2224,7 +2189,7 @@ dependencies = [
"datafusion-functions-window-common",
"datafusion-physical-expr-common",
"env_logger",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
"itertools 0.14.0",
"paste",
@@ -2239,7 +2204,7 @@ version = "50.0.0"
dependencies = [
"arrow",
"datafusion-common",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"itertools 0.14.0",
"paste",
]
@@ -2260,7 +2225,7 @@ dependencies = [
"doc-comment",
"futures",
"log",
- "prost",
+ "prost 0.13.5",
"semver",
"tokio",
]
@@ -2395,7 +2360,7 @@ dependencies = [
name = "datafusion-macros"
version = "50.0.0"
dependencies = [
- "datafusion-expr",
+ "datafusion-doc",
"quote",
"syn 2.0.106",
]
@@ -2418,7 +2383,7 @@ dependencies = [
"datafusion-physical-expr",
"datafusion-sql",
"env_logger",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
"itertools 0.14.0",
"log",
@@ -2442,7 +2407,7 @@ dependencies = [
"datafusion-physical-expr-common",
"half",
"hashbrown 0.14.5",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
"itertools 0.14.0",
"parking_lot",
@@ -2520,7 +2485,7 @@ dependencies = [
"futures",
"half",
"hashbrown 0.14.5",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
"itertools 0.14.0",
"log",
@@ -2549,7 +2514,7 @@ dependencies = [
"object_store",
"pbjson",
"pretty_assertions",
- "prost",
+ "prost 0.13.5",
"serde",
"serde_json",
"tokio",
@@ -2563,7 +2528,7 @@ dependencies = [
"datafusion-common",
"doc-comment",
"pbjson",
- "prost",
+ "prost 0.13.5",
"serde",
]
@@ -2622,6 +2587,7 @@ version = "50.0.0"
dependencies = [
"arrow",
"bigdecimal",
+ "chrono",
"ctor",
"datafusion-common",
"datafusion-expr",
@@ -2630,8 +2596,9 @@ dependencies = [
"datafusion-functions-nested",
"datafusion-functions-window",
"env_logger",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"insta",
+ "itertools 0.14.0",
"log",
"paste",
"recursive",
@@ -2649,7 +2616,7 @@ dependencies = [
"bigdecimal",
"bytes",
"chrono",
- "clap 4.5.47",
+ "clap 4.5.48",
"datafusion",
"datafusion-spark",
"datafusion-substrait",
@@ -2687,7 +2654,7 @@ dependencies = [
"itertools 0.14.0",
"object_store",
"pbjson-types",
- "prost",
+ "prost 0.13.5",
"serde_json",
"substrait",
"tokio",
@@ -2718,9 +2685,9 @@ dependencies = [
[[package]]
name = "deranged"
-version = "0.4.0"
+version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c9e6a11ca8224451684bc0d7d5a7adbf8f2fd6887261a1cfc3c0432f9d4068e"
+checksum = "d630bccd429a5bb5a64b5e94f693bfc48c9f8566418fda4c494cc94f911f87cc"
dependencies = [
"powerfmt",
"serde",
@@ -2761,7 +2728,7 @@ dependencies = [
"libc",
"option-ext",
"redox_users",
- "windows-sys 0.60.2",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -2900,12 +2867,12 @@ checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "errno"
-version = "0.3.13"
+version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "778e2ac28f6c47af28e4907f13ffd1e1ddbd400980a9abd7c8df189bf578a5ad"
+checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
- "windows-sys 0.60.2",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -2950,7 +2917,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ce92ff622d6dadf7349484f42c93271a0d49b7cc4d466a936405bacbe10aa78"
dependencies = [
"cfg-if",
- "rustix 1.0.8",
+ "rustix",
"windows-sys 0.59.0",
]
@@ -2986,16 +2953,22 @@ dependencies = [
[[package]]
name = "filetime"
-version = "0.2.25"
+version = "0.2.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "35c0522e981e68cbfa8c3f978441a5f34b30b96e146b33cd3359176b50fe8586"
+checksum = "bc0505cd1b6fa6580283f6bdf70a73fcf4aba1184038c90902b92b3dd0df63ed"
dependencies = [
"cfg-if",
"libc",
"libredox",
- "windows-sys 0.59.0",
+ "windows-sys 0.60.2",
]
+[[package]]
+name = "find-msvc-tools"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1ced73b1dacfc750a6db6c0a0c3a3853c8b41997e2e2c563dc90804ae6867959"
+
[[package]]
name = "fixedbitset"
version = "0.5.7"
@@ -3008,7 +2981,7 @@ version = "25.2.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1045398c1bfd89168b5fd3f1fc11f6e70b34f6f66300c87d44d3de849463abf1"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"rustc_version",
]
@@ -3046,9 +3019,9 @@ dependencies = [
[[package]]
name = "fs-err"
-version = "3.1.1"
+version = "3.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "88d7be93788013f265201256d58f04936a8079ad5dc898743aa20525f503b683"
+checksum = "44f150ffc8782f35521cec2b23727707cb4045706ba3c854e86bef66b3a8cdbd"
dependencies = [
"autocfg",
]
@@ -3164,16 +3137,16 @@ dependencies = [
name = "gen"
version = "0.1.0"
dependencies = [
- "pbjson-build",
- "prost-build",
+ "pbjson-build 0.8.0",
+ "prost-build 0.14.1",
]
[[package]]
name = "gen-common"
version = "0.1.0"
dependencies = [
- "pbjson-build",
- "prost-build",
+ "pbjson-build 0.8.0",
+ "prost-build 0.14.1",
]
[[package]]
@@ -3218,7 +3191,7 @@ dependencies = [
"js-sys",
"libc",
"r-efi",
- "wasi 0.14.2+wasi-0.2.4",
+ "wasi 0.14.7+wasi-0.2.4",
"wasm-bindgen",
]
@@ -3259,7 +3232,7 @@ dependencies = [
"futures-core",
"futures-sink",
"http 1.3.1",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"slab",
"tokio",
"tokio-util",
@@ -3307,6 +3280,12 @@ dependencies = [
"foldhash",
]
+[[package]]
+name = "hashbrown"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d"
+
[[package]]
name = "heck"
version = "0.3.3"
@@ -3422,19 +3401,20 @@ checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
[[package]]
name = "humantime"
-version = "2.2.0"
+version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9b112acc8b3adf4b107a8ec20977da0273a8c386765a3ec0229bd500a1443f9f"
+checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424"
[[package]]
name = "hyper"
-version = "1.6.0"
+version = "1.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cc2b571658e38e0c01b1fdca3bbbe93c00d3d71693ff2770043f8c29bc7d6f80"
+checksum = "eb3aa54a13a0dfe7fbe3a59e0c76093041720fdc77b110cc0fc260fafb4dc51e"
dependencies = [
+ "atomic-waker",
"bytes",
"futures-channel",
- "futures-util",
+ "futures-core",
"h2",
"http 1.3.1",
"http-body 1.0.1",
@@ -3442,6 +3422,7 @@ dependencies = [
"httpdate",
"itoa",
"pin-project-lite",
+ "pin-utils",
"smallvec",
"tokio",
"want",
@@ -3494,9 +3475,9 @@ dependencies = [
[[package]]
name = "hyper-util"
-version = "0.1.16"
+version = "0.1.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8d9b05277c7e8da2c93a568989bb6207bef0112e8d17df7a6eda4a3cf143bc5e"
+checksum = "3c6995591a8f1380fcb4ba966a252a4b29188d51d2b89e3a252f5305be65aea8"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -3533,9 +3514,9 @@ dependencies = [
[[package]]
name = "iana-time-zone"
-version = "0.1.63"
+version = "0.1.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0c919e5debc312ad217002b8048a17b7d83f80703865bbfcfebb0458b0b27d8"
+checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb"
dependencies = [
"android_system_properties",
"core-foundation-sys",
@@ -3543,7 +3524,7 @@ dependencies = [
"js-sys",
"log",
"wasm-bindgen",
- "windows-core",
+ "windows-core 0.62.0",
]
[[package]]
@@ -3681,12 +3662,12 @@ dependencies = [
[[package]]
name = "indexmap"
-version = "2.11.3"
+version = "2.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "92119844f513ffa41556430369ab02c295a3578af21cf945caa3e9e0c2481ac3"
+checksum = "4b0f83760fb341a774ed326568e19f5a863af4a952def8c39f9ab92fd95b88e5"
dependencies = [
"equivalent",
- "hashbrown 0.15.5",
+ "hashbrown 0.16.0",
"serde",
"serde_core",
]
@@ -3697,7 +3678,7 @@ version = "0.18.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70a646d946d06bedbbc4cac4c218acf4bbf2d87757a784857025f4d447e4e1cd"
dependencies = [
- "console 0.16.0",
+ "console 0.16.1",
"portable-atomic",
"unicode-width 0.2.1",
"unit-prefix",
@@ -3744,11 +3725,11 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02"
[[package]]
name = "io-uring"
-version = "0.7.9"
+version = "0.7.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d93587f37623a1a17d94ef2bc9ada592f5465fe7732084ab7beefabe5c77c0c4"
+checksum = "046fa2d4d00aea763528b4950358d0ead425372445dc8ff86312b3c69ff7727b"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"cfg-if",
"libc",
]
@@ -3795,15 +3776,6 @@ dependencies = [
"either",
]
-[[package]]
-name = "itertools"
-version = "0.12.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569"
-dependencies = [
- "either",
-]
-
[[package]]
name = "itertools"
version = "0.13.0"
@@ -3854,9 +3826,9 @@ dependencies = [
[[package]]
name = "jobserver"
-version = "0.1.33"
+version = "0.1.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "38f262f097c174adebe41eb73d66ae9c06b2844fb0da69969647bbddd9b0538a"
+checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
dependencies = [
"getrandom 0.3.3",
"libc",
@@ -3864,9 +3836,9 @@ dependencies = [
[[package]]
name = "js-sys"
-version = "0.3.78"
+version = "0.3.81"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0c0b063578492ceec17683ef2f8c5e89121fbd0b172cbc280635ab7567db2738"
+checksum = "ec48937a97411dcb524a265206ccd4c90bb711fca92b2792c407f268825b9305"
dependencies = [
"once_cell",
"wasm-bindgen",
@@ -3878,17 +3850,11 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
-[[package]]
-name = "lazycell"
-version = "1.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"
-
[[package]]
name = "lexical-core"
-version = "1.0.5"
+version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b765c31809609075565a70b4b71402281283aeda7ecaf4818ac14a7b2ade8958"
+checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594"
dependencies = [
"lexical-parse-float",
"lexical-parse-integer",
@@ -3899,53 +3865,46 @@ dependencies = [
[[package]]
name = "lexical-parse-float"
-version = "1.0.5"
+version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "de6f9cb01fb0b08060209a057c048fcbab8717b4c1ecd2eac66ebfe39a65b0f2"
+checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56"
dependencies = [
"lexical-parse-integer",
"lexical-util",
- "static_assertions",
]
[[package]]
name = "lexical-parse-integer"
-version = "1.0.5"
+version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "72207aae22fc0a121ba7b6d479e42cbfea549af1479c3f3a4f12c70dd66df12e"
+checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34"
dependencies = [
"lexical-util",
- "static_assertions",
]
[[package]]
name = "lexical-util"
-version = "1.0.6"
+version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5a82e24bf537fd24c177ffbbdc6ebcc8d54732c35b50a3f28cc3f4e4c949a0b3"
-dependencies = [
- "static_assertions",
-]
+checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17"
[[package]]
name = "lexical-write-float"
-version = "1.0.5"
+version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c5afc668a27f460fb45a81a757b6bf2f43c2d7e30cb5a2dcd3abf294c78d62bd"
+checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361"
dependencies = [
"lexical-util",
"lexical-write-integer",
- "static_assertions",
]
[[package]]
name = "lexical-write-integer"
-version = "1.0.5"
+version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "629ddff1a914a836fb245616a7888b62903aae58fa771e1d83943035efa0f978"
+checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df"
dependencies = [
"lexical-util",
- "static_assertions",
]
[[package]]
@@ -3956,9 +3915,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7"
[[package]]
name = "libc"
-version = "0.2.175"
+version = "0.2.176"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a82ae493e598baaea5209805c49bbf2ea7de956d50d7da0da1164f9c6d28543"
+checksum = "58f929b4d672ea937a23a1ab494143d968337a5f47e56d0815df1e0890ddf174"
[[package]]
name = "libloading"
@@ -3972,12 +3931,12 @@ dependencies = [
[[package]]
name = "libloading"
-version = "0.8.8"
+version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "07033963ba89ebaf1584d767badaa2e8fcec21aedea6b8c0346d487d49c28667"
+checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
dependencies = [
"cfg-if",
- "windows-targets 0.53.3",
+ "windows-link 0.2.0",
]
[[package]]
@@ -3999,11 +3958,11 @@ dependencies = [
[[package]]
name = "libredox"
-version = "0.1.9"
+version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "391290121bad3d37fbddad76d8f5d1c1c314cfc646d143d7e07a3086ddff0ce3"
+checksum = "416f7e718bdb06000964960ffa43b4335ad4012ae8b99060261aa4a8088d5ccb"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"libc",
"redox_syscall 0.5.17",
]
@@ -4016,30 +3975,24 @@ checksum = "5297962ef19edda4ce33aaa484386e0a5b3d7f2f4e037cbeee00503ef6b29d33"
dependencies = [
"anstream",
"anstyle",
- "clap 4.5.47",
+ "clap 4.5.48",
"escape8259",
]
[[package]]
name = "libz-rs-sys"
-version = "0.5.1"
+version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "172a788537a2221661b480fee8dc5f96c580eb34fa88764d3205dc356c7e4221"
+checksum = "840db8cf39d9ec4dd794376f38acc40d0fc65eec2a8f484f7fd375b84602becd"
dependencies = [
"zlib-rs",
]
[[package]]
name = "linux-raw-sys"
-version = "0.4.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d26c52dbd32dccf2d10cac7725f8eae5296885fb5703b261f7d0a0739ec807ab"
-
-[[package]]
-name = "linux-raw-sys"
-version = "0.9.4"
+version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cd945864f07fe9f5371a27ad7b52a172b4b499999f1d97574c9fa68373937e12"
+checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039"
[[package]]
name = "litemap"
@@ -4192,7 +4145,7 @@ version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"cfg-if",
"cfg_aliases",
"libc",
@@ -4313,7 +4266,7 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c10c2894a6fed806ade6027bcd50662746363a9589d3ec9d9bef30a4e4bc166"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
]
[[package]]
@@ -4337,9 +4290,9 @@ dependencies = [
[[package]]
name = "object_store"
-version = "0.12.3"
+version = "0.12.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "efc4f07659e11cd45a341cd24d71e683e3be65d9ff1f8150061678fe60437496"
+checksum = "4c1be0c6c22ec0817cdc77d3842f721a17fd30ab6965001415b5402a74e6b740"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -4448,9 +4401,9 @@ dependencies = [
[[package]]
name = "parquet"
-version = "56.0.0"
+version = "56.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c7288a07ed5d25939a90f9cb1ca5afa6855faa08ec7700613511ae64bdb0620c"
+checksum = "f0dbd48ad52d7dccf8ea1b90a3ddbfaea4f69878dd7683e51c507d4bc52b5b27"
dependencies = [
"ahash 0.8.12",
"arrow-array",
@@ -4467,7 +4420,7 @@ dependencies = [
"flate2",
"futures",
"half",
- "hashbrown 0.15.5",
+ "hashbrown 0.16.0",
"lz4_flex",
"num",
"num-bigint",
@@ -4532,8 +4485,20 @@ checksum = "6eea3058763d6e656105d1403cb04e0a41b7bbac6362d413e7c33be0c32279c9"
dependencies = [
"heck 0.5.0",
"itertools 0.13.0",
- "prost",
- "prost-types",
+ "prost 0.13.5",
+ "prost-types 0.13.5",
+]
+
+[[package]]
+name = "pbjson-build"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "af22d08a625a2213a78dbb0ffa253318c5c79ce3133d32d296655a7bdfb02095"
+dependencies = [
+ "heck 0.5.0",
+ "itertools 0.14.0",
+ "prost 0.14.1",
+ "prost-types 0.14.1",
]
[[package]]
@@ -4545,9 +4510,9 @@ dependencies = [
"bytes",
"chrono",
"pbjson",
- "pbjson-build",
- "prost",
- "prost-build",
+ "pbjson-build 0.7.0",
+ "prost 0.13.5",
+ "prost-build 0.13.5",
"serde",
]
@@ -4564,7 +4529,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
dependencies = [
"fixedbitset",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
]
[[package]]
@@ -4575,42 +4540,43 @@ checksum = "54acf3a685220b533e437e264e4d932cfbdc4cc7ec0cd232ed73c08d03b8a7ca"
dependencies = [
"fixedbitset",
"hashbrown 0.15.5",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"serde",
]
[[package]]
name = "phf"
-version = "0.11.3"
+version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078"
+checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7"
dependencies = [
- "phf_shared 0.11.3",
+ "phf_shared 0.12.1",
]
[[package]]
name = "phf"
-version = "0.12.1"
+version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7"
+checksum = "c1562dc717473dbaa4c1f85a36410e03c047b2e7df7f45ee938fbef64ae7fadf"
dependencies = [
- "phf_shared 0.12.1",
+ "phf_shared 0.13.1",
+ "serde",
]
[[package]]
name = "phf_shared"
-version = "0.11.3"
+version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5"
+checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981"
dependencies = [
"siphasher",
]
[[package]]
name = "phf_shared"
-version = "0.12.1"
+version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981"
+checksum = "e57fef6bc5981e38c2ce2d63bfa546861309f875b8a75f092d1d54ae2d64f266"
dependencies = [
"siphasher",
]
@@ -4698,9 +4664,9 @@ dependencies = [
[[package]]
name = "postgres-derive"
-version = "0.4.6"
+version = "0.4.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "69700ea4603c5ef32d447708e6a19cd3e8ac197a000842e97f527daea5e4175f"
+checksum = "56df96f5394370d1b20e49de146f9e6c25aa9ae750f449c9d665eafecb3ccae6"
dependencies = [
"heck 0.5.0",
"proc-macro2",
@@ -4710,9 +4676,9 @@ dependencies = [
[[package]]
name = "postgres-protocol"
-version = "0.6.8"
+version = "0.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "76ff0abab4a9b844b93ef7b81f1efc0a366062aaef2cd702c76256b5dc075c54"
+checksum = "fbef655056b916eb868048276cfd5d6a7dea4f81560dfd047f97c8c6fe3fcfd4"
dependencies = [
"base64 0.22.1",
"byteorder",
@@ -4728,9 +4694,9 @@ dependencies = [
[[package]]
name = "postgres-types"
-version = "0.2.9"
+version = "0.2.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "613283563cd90e1dfc3518d548caee47e0e725455ed619881f5cf21f36de4b48"
+checksum = "77a120daaabfcb0e324d5bf6e411e9222994cb3795c79943a0ef28ed27ea76e4"
dependencies = [
"bytes",
"chrono",
@@ -4741,9 +4707,9 @@ dependencies = [
[[package]]
name = "potential_utf"
-version = "0.1.2"
+version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5a7c30837279ca13e7c867e9e40053bc68740f988cb07f7ca6df43cc734b585"
+checksum = "84df19adbe5b5a0782edcab45899906947ab039ccf4573713735ee7de1e6b08a"
dependencies = [
"zerovec",
]
@@ -4775,9 +4741,9 @@ dependencies = [
[[package]]
name = "prettyplease"
-version = "0.2.36"
+version = "0.2.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ff24dfcda44452b9816fff4cd4227e1bb73ff5a2f1bc1105aa92fb8565ce44d2"
+checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
dependencies = [
"proc-macro2",
"syn 2.0.106",
@@ -4785,9 +4751,9 @@ dependencies = [
[[package]]
name = "proc-macro-crate"
-version = "3.3.0"
+version = "3.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "edce586971a4dfaa28950c6f18ed55e0406c1ab88bbce2c6f6293a7aaba73d35"
+checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983"
dependencies = [
"toml_edit",
]
@@ -4818,9 +4784,9 @@ dependencies = [
[[package]]
name = "proc-macro2"
-version = "1.0.97"
+version = "1.0.101"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d61789d7719defeb74ea5fe81f2fdfdbd28a803847077cecce2ff14e1472f6f1"
+checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de"
dependencies = [
"unicode-ident",
]
@@ -4832,7 +4798,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5"
dependencies = [
"bytes",
- "prost-derive",
+ "prost-derive 0.13.5",
+]
+
+[[package]]
+name = "prost"
+version = "0.14.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7231bd9b3d3d33c86b58adbac74b5ec0ad9f496b19d22801d773636feaa95f3d"
+dependencies = [
+ "bytes",
+ "prost-derive 0.14.1",
]
[[package]]
@@ -4848,8 +4824,28 @@ dependencies = [
"once_cell",
"petgraph 0.7.1",
"prettyplease",
- "prost",
- "prost-types",
+ "prost 0.13.5",
+ "prost-types 0.13.5",
+ "regex",
+ "syn 2.0.106",
+ "tempfile",
+]
+
+[[package]]
+name = "prost-build"
+version = "0.14.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac6c3320f9abac597dcbc668774ef006702672474aad53c6d596b62e487b40b1"
+dependencies = [
+ "heck 0.5.0",
+ "itertools 0.14.0",
+ "log",
+ "multimap",
+ "once_cell",
+ "petgraph 0.7.1",
+ "prettyplease",
+ "prost 0.14.1",
+ "prost-types 0.14.1",
"regex",
"syn 2.0.106",
"tempfile",
@@ -4868,13 +4864,35 @@ dependencies = [
"syn 2.0.106",
]
+[[package]]
+name = "prost-derive"
+version = "0.14.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9120690fafc389a67ba3803df527d0ec9cbbc9cc45e4cc20b332996dfb672425"
+dependencies = [
+ "anyhow",
+ "itertools 0.14.0",
+ "proc-macro2",
+ "quote",
+ "syn 2.0.106",
+]
+
[[package]]
name = "prost-types"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16"
dependencies = [
- "prost",
+ "prost 0.13.5",
+]
+
+[[package]]
+name = "prost-types"
+version = "0.14.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9b4db3d6da204ed77bb26ba83b6122a73aeb2e87e25fbf7ad2e84c4ccbf8f72"
+dependencies = [
+ "prost 0.14.1",
]
[[package]]
@@ -4985,9 +5003,9 @@ checksum = "5a651516ddc9168ebd67b24afd085a718be02f8858fe406591b013d101ce2f40"
[[package]]
name = "quick-xml"
-version = "0.38.1"
+version = "0.38.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9845d9dccf565065824e69f9f235fafba1587031eda353c1f1561cd6a6be78f4"
+checksum = "42a232e7487fc2ef313d96dde7948e7a3c05101870d8985e4fd8d26aedd27b89"
dependencies = [
"memchr",
"serde",
@@ -4995,18 +5013,18 @@ dependencies = [
[[package]]
name = "quinn"
-version = "0.11.8"
+version = "0.11.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "626214629cda6781b6dc1d316ba307189c85ba657213ce642d9c77670f8202c8"
+checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
dependencies = [
"bytes",
"cfg_aliases",
"pin-project-lite",
"quinn-proto",
"quinn-udp",
- "rustc-hash 2.1.1",
+ "rustc-hash",
"rustls",
- "socket2 0.5.10",
+ "socket2 0.6.0",
"thiserror",
"tokio",
"tracing",
@@ -5015,16 +5033,16 @@ dependencies = [
[[package]]
name = "quinn-proto"
-version = "0.11.12"
+version = "0.11.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "49df843a9161c85bb8aae55f101bc0bac8bcafd637a620d9122fd7e0b2f7422e"
+checksum = "f1906b49b0c3bc04b5fe5d86a77925ae6524a19b816ae38ce1e426255f1d8a31"
dependencies = [
"bytes",
"getrandom 0.3.3",
"lru-slab",
"rand 0.9.2",
"ring",
- "rustc-hash 2.1.1",
+ "rustc-hash",
"rustls",
"rustls-pki-types",
"slab",
@@ -5036,23 +5054,23 @@ dependencies = [
[[package]]
name = "quinn-udp"
-version = "0.5.13"
+version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fcebb1209ee276352ef14ff8732e24cc2b02bbac986cd74a4c81bcb2f9881970"
+checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
dependencies = [
"cfg_aliases",
"libc",
"once_cell",
- "socket2 0.5.10",
+ "socket2 0.6.0",
"tracing",
- "windows-sys 0.59.0",
+ "windows-sys 0.60.2",
]
[[package]]
name = "quote"
-version = "1.0.40"
+version = "1.0.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1885c039570dc00dcb4ff087a89e185fd56bae234ddc7f056a945bf36467248d"
+checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1"
dependencies = [
"proc-macro2",
]
@@ -5203,7 +5221,7 @@ version = "0.5.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5407465600fb0548f1442edf71dd20683c6ed326200ace4b1ef0763521bb3b77"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
]
[[package]]
@@ -5239,9 +5257,9 @@ dependencies = [
[[package]]
name = "regex"
-version = "1.11.2"
+version = "1.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "23d7fd106d8c02486a8d64e778353d1cffe08ce79ac2e82f540c86d0facf6912"
+checksum = "8b5288124840bee7b386bc413c487869b360b2b4ec421ea56425128692f2a82c"
dependencies = [
"aho-corasick",
"memchr",
@@ -5251,9 +5269,9 @@ dependencies = [
[[package]]
name = "regex-automata"
-version = "0.4.9"
+version = "0.4.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908"
+checksum = "833eb9ce86d40ef33cb1306d8accf7bc8ec2bfea4355cbdebb3df68b40925cad"
dependencies = [
"aho-corasick",
"memchr",
@@ -5262,9 +5280,9 @@ dependencies = [
[[package]]
name = "regex-lite"
-version = "0.1.6"
+version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "53a49587ad06b26609c52e423de037e7f57f20d53535d66e08c695f347df952a"
+checksum = "943f41321c63ef1c92fd763bfe054d2668f7f225a5c29f0105903dc2fc04ba30"
[[package]]
name = "regex-syntax"
@@ -5455,12 +5473,6 @@ version = "0.1.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56f7d92ca342cea22a06f2121d944b4fd82af56988c270852495420f961d4ace"
-[[package]]
-name = "rustc-hash"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
-
[[package]]
name = "rustc-hash"
version = "2.1.1"
@@ -5478,35 +5490,22 @@ dependencies = [
[[package]]
name = "rustix"
-version = "0.38.44"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154"
-dependencies = [
- "bitflags 2.9.1",
- "errno",
- "libc",
- "linux-raw-sys 0.4.15",
- "windows-sys 0.59.0",
-]
-
-[[package]]
-name = "rustix"
-version = "1.0.8"
+version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "11181fbabf243db407ef8df94a6ce0b2f9a733bd8be4ad02b4eda9602296cac8"
+checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"errno",
"libc",
- "linux-raw-sys 0.9.4",
- "windows-sys 0.60.2",
+ "linux-raw-sys",
+ "windows-sys 0.61.0",
]
[[package]]
name = "rustls"
-version = "0.23.31"
+version = "0.23.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c0ebcbd2f03de0fc1122ad9bb24b127a5a6cd51d72604a3f3c50ac459762b6cc"
+checksum = "cd3c25631629d034ce7cd9940adc9d45762d46de2b0f57193c4443b92c6d4d40"
dependencies = [
"aws-lc-rs",
"once_cell",
@@ -5550,9 +5549,9 @@ dependencies = [
[[package]]
name = "rustls-webpki"
-version = "0.103.4"
+version = "0.103.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0a17884ae0c1b773f1ccd2bd4a8c72f16da897310a98b0e84bf349ad5ead92fc"
+checksum = "8572f3c2cb9934231157b45499fc41e1f58c589fdfb81a844ba873265e80f8eb"
dependencies = [
"aws-lc-rs",
"ring",
@@ -5572,7 +5571,7 @@ version = "17.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6614df0b6d4cfb20d1d5e295332921793ce499af3ebc011bf1e393380e1e492"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"cfg-if",
"clipboard-win",
"fd-lock",
@@ -5605,11 +5604,11 @@ dependencies = [
[[package]]
name = "schannel"
-version = "0.1.27"
+version = "0.1.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1f29ebaa345f945cec9fbbc532eb307f0fdad8161f281b6369539c8d84876b3d"
+checksum = "891d81b926048e76efe18581bf793546b4c0eaf8448d72be8de2bbee5fd166e1"
dependencies = [
- "windows-sys 0.59.0",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -5674,11 +5673,11 @@ checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b"
[[package]]
name = "security-framework"
-version = "3.3.0"
+version = "3.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "80fb1d92c5028aa318b4b8bd7302a5bfcf48be96a37fc6fc790f806b0004ee0c"
+checksum = "cc198e42d9b7510827939c9a15f5062a0c913f3371d765977e586d2fe6c16f4a"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"core-foundation",
"core-foundation-sys",
"libc",
@@ -5687,9 +5686,9 @@ dependencies = [
[[package]]
name = "security-framework-sys"
-version = "2.14.0"
+version = "2.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "49db231d56a190491cb4aeda9527f1ad45345af50b0851622a7adb8c03b01c32"
+checksum = "cc1f0cbffaac4852523ce30d8bd3c5cdc873501d96ff467ca09b6767bb8cd5c0"
dependencies = [
"core-foundation-sys",
"libc",
@@ -5713,9 +5712,9 @@ checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
[[package]]
name = "serde"
-version = "1.0.223"
+version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a505d71960adde88e293da5cb5eda57093379f64e61cf77bf0e6a63af07a7bac"
+checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
@@ -5723,27 +5722,28 @@ dependencies = [
[[package]]
name = "serde_bytes"
-version = "0.11.17"
+version = "0.11.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8437fd221bde2d4ca316d61b90e337e9e702b3820b87d63caa9ba6c02bd06d96"
+checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8"
dependencies = [
"serde",
+ "serde_core",
]
[[package]]
name = "serde_core"
-version = "1.0.223"
+version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20f57cbd357666aa7b3ac84a90b4ea328f1d4ddb6772b430caa5d9e1309bb9e9"
+checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
-version = "1.0.223"
+version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3d428d07faf17e306e699ec1e91996e5a165ba5d6bce5b5155173e91a8a01a56"
+checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
@@ -5811,15 +5811,15 @@ dependencies = [
[[package]]
name = "serde_with"
-version = "3.14.0"
+version = "3.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f2c45cd61fefa9db6f254525d46e392b852e0e61d9a1fd36e5bd183450a556d5"
+checksum = "c522100790450cf78eeac1507263d0a350d4d5b30df0c8e1fe051a10c22b376e"
dependencies = [
"base64 0.22.1",
"chrono",
"hex",
"indexmap 1.9.3",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"schemars 0.9.0",
"schemars 1.0.4",
"serde",
@@ -5831,11 +5831,11 @@ dependencies = [
[[package]]
name = "serde_with_macros"
-version = "3.14.0"
+version = "3.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "de90945e6565ce0d9a25098082ed4ee4002e047cb59892c318d66821e14bb30f"
+checksum = "327ada00f7d64abaac1e55a6911e90cf665aa051b9a561c7006c157f4633135e"
dependencies = [
- "darling 0.20.11",
+ "darling",
"proc-macro2",
"quote",
"syn 2.0.106",
@@ -5847,7 +5847,7 @@ version = "0.9.34+deprecated"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
dependencies = [
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"itoa",
"ryu",
"serde",
@@ -6040,12 +6040,6 @@ dependencies = [
"windows-sys 0.59.0",
]
-[[package]]
-name = "static_assertions"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
-
[[package]]
name = "stringprep"
version = "0.1.5"
@@ -6110,12 +6104,31 @@ dependencies = [
"syn 1.0.109",
]
+[[package]]
+name = "strum"
+version = "0.26.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06"
+
[[package]]
name = "strum"
version = "0.27.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf"
+[[package]]
+name = "strum_macros"
+version = "0.26.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be"
+dependencies = [
+ "heck 0.5.0",
+ "proc-macro2",
+ "quote",
+ "rustversion",
+ "syn 2.0.106",
+]
+
[[package]]
name = "strum_macros"
version = "0.27.2"
@@ -6146,12 +6159,12 @@ checksum = "de6d24c270c6c672a86c183c3a8439ba46c1936f93cf7296aa692de3b0ff0228"
dependencies = [
"heck 0.5.0",
"pbjson",
- "pbjson-build",
+ "pbjson-build 0.7.0",
"pbjson-types",
"prettyplease",
- "prost",
- "prost-build",
- "prost-types",
+ "prost 0.13.5",
+ "prost-build 0.13.5",
+ "prost-types 0.13.5",
"protobuf-src",
"regress",
"schemars 0.8.22",
@@ -6214,9 +6227,9 @@ dependencies = [
[[package]]
name = "sysinfo"
-version = "0.37.0"
+version = "0.37.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "07cec4dc2d2e357ca1e610cfb07de2fa7a10fc3e9fe89f72545f3d244ea87753"
+checksum = "3bddd368fda2f82ead69c03d46d351987cfa0c2a57abfa37a017f3aa3e9bf69a"
dependencies = [
"libc",
"memchr",
@@ -6234,21 +6247,21 @@ checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369"
[[package]]
name = "target-lexicon"
-version = "0.13.2"
+version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e502f78cdbb8ba4718f566c418c52bc729126ffd16baee5baa718cf25dd5a69a"
+checksum = "df7f62577c25e07834649fc3b39fafdc597c0a3527dc1c60129201ccfcbaa50c"
[[package]]
name = "tempfile"
-version = "3.22.0"
+version = "3.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "84fa4d11fadde498443cca10fd3ac23c951f0dc59e080e9f4b93d4df4e4eea53"
+checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16"
dependencies = [
"fastrand",
"getrandom 0.3.3",
"once_cell",
- "rustix 1.0.8",
- "windows-sys 0.60.2",
+ "rustix",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -6311,18 +6324,18 @@ dependencies = [
[[package]]
name = "thiserror"
-version = "2.0.16"
+version = "2.0.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3467d614147380f2e4e374161426ff399c91084acd2363eaf549172b3d5e60c0"
+checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
-version = "2.0.16"
+version = "2.0.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c5e1be1c48b9172ee610da68fd9cd2770e7a4056cb3fc98710ee6906f0c7960"
+checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913"
dependencies = [
"proc-macro2",
"quote",
@@ -6351,9 +6364,9 @@ dependencies = [
[[package]]
name = "time"
-version = "0.3.41"
+version = "0.3.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8a7619e19bc266e0f9c5e6686659d394bc57973859340060a69221e57dbc0c40"
+checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d"
dependencies = [
"deranged",
"itoa",
@@ -6366,15 +6379,15 @@ dependencies = [
[[package]]
name = "time-core"
-version = "0.1.4"
+version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c9e9a38711f559d9e3ce1cdb06dd7c5b8ea546bc90052da6d06bb76da74bb07c"
+checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b"
[[package]]
name = "time-macros"
-version = "0.2.22"
+version = "0.2.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3526739392ec93fd8b359c8e98514cb3e8e021beb4e5f597b00a0221f8ed8a49"
+checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3"
dependencies = [
"num-conv",
"time-core",
@@ -6411,9 +6424,9 @@ dependencies = [
[[package]]
name = "tinyvec"
-version = "1.9.0"
+version = "1.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "09b3661f17e86524eccd4371ab0429194e0d7c008abb45f7a7495b1719463c71"
+checksum = "bfa5fdc3bce6191a1dbc8c02d5c8bffcf557bafa17c124c5264a458f1b0613fa"
dependencies = [
"tinyvec_macros",
]
@@ -6457,9 +6470,9 @@ dependencies = [
[[package]]
name = "tokio-postgres"
-version = "0.7.13"
+version = "0.7.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c95d533c83082bb6490e0189acaa0bbeef9084e60471b696ca6988cd0541fb0"
+checksum = "a156efe7fff213168257853e1dfde202eed5f487522cbbbf7d219941d753d853"
dependencies = [
"async-trait",
"byteorder",
@@ -6470,12 +6483,12 @@ dependencies = [
"log",
"parking_lot",
"percent-encoding",
- "phf 0.11.3",
+ "phf 0.13.1",
"pin-project-lite",
"postgres-protocol",
"postgres-types",
"rand 0.9.2",
- "socket2 0.5.10",
+ "socket2 0.6.0",
"tokio",
"tokio-util",
"whoami",
@@ -6483,9 +6496,9 @@ dependencies = [
[[package]]
name = "tokio-rustls"
-version = "0.26.2"
+version = "0.26.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8e727b36a1a0e8b74c376ac2211e40c2c8af09fb4013c60d910495810f008e9b"
+checksum = "05f63835928ca123f1bef57abbcd23bb2ba0ac9ae1235f1e65bda0d06e7786bd"
dependencies = [
"rustls",
"tokio",
@@ -6532,18 +6545,31 @@ dependencies = [
[[package]]
name = "toml_datetime"
-version = "0.6.11"
+version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
+checksum = "32f1085dec27c2b6632b04c80b3bb1b4300d6495d1e129693bdda7d91e72eec1"
+dependencies = [
+ "serde_core",
+]
[[package]]
name = "toml_edit"
-version = "0.22.27"
+version = "0.23.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
+checksum = "f3effe7c0e86fdff4f69cdd2ccc1b96f933e24811c5441d44904e8683e27184b"
dependencies = [
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"toml_datetime",
+ "toml_parser",
+ "winnow",
+]
+
+[[package]]
+name = "toml_parser"
+version = "1.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4cf893c33be71572e0e9aa6dd15e6677937abd686b066eac3f8cd3531688a627"
+dependencies = [
"winnow",
]
@@ -6566,7 +6592,7 @@ dependencies = [
"hyper-util",
"percent-encoding",
"pin-project",
- "prost",
+ "prost 0.13.5",
"socket2 0.5.10",
"tokio",
"tokio-stream",
@@ -6584,7 +6610,7 @@ checksum = "d039ad9159c98b70ecfd540b2573b97f7f52c3e8d9f8ad57a24b916a536975f9"
dependencies = [
"futures-core",
"futures-util",
- "indexmap 2.11.3",
+ "indexmap 2.11.4",
"pin-project-lite",
"slab",
"sync_wrapper",
@@ -6601,7 +6627,7 @@ version = "0.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "adc82fd73de2a9722ac5da747f12383d2bfdb93591ee6c58486e0097890f05f2"
dependencies = [
- "bitflags 2.9.1",
+ "bitflags 2.9.4",
"bytes",
"futures-util",
"http 1.3.1",
@@ -6705,9 +6731,9 @@ checksum = "e78122066b0cb818b8afd08f7ed22f7fdbc3e90815035726f0840d0d26c0747a"
[[package]]
name = "twox-hash"
-version = "2.1.1"
+version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8b907da542cbced5261bd3256de1b3a1bf340a3d37f93425a07362a1d687de56"
+checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c"
[[package]]
name = "typed-arena"
@@ -6723,15 +6749,15 @@ checksum = "1dccffe3ce07af9386bfd29e80c0ab1a8205a2fc34e4bcd40364df902cfa8f3f"
[[package]]
name = "typewit"
-version = "1.12.1"
+version = "1.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "97e72ba082eeb9da9dc68ff5a2bf727ef6ce362556e8d29ec1aed3bd05e7d86a"
+checksum = "f8c1ae7cc0fdb8b842d65d127cb981574b0d2b249b74d1c7a2986863dc134f71"
[[package]]
name = "typify"
-version = "0.4.2"
+version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6c6c647a34e851cf0260ccc14687f17cdcb8302ff1a8a687a24b97ca0f82406f"
+checksum = "7144144e97e987c94758a3017c920a027feac0799df325d6df4fc8f08d02068e"
dependencies = [
"typify-impl",
"typify-macro",
@@ -6739,9 +6765,9 @@ dependencies = [
[[package]]
name = "typify-impl"
-version = "0.4.2"
+version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "741b7f1e2e1338c0bee5ad5a7d3a9bbd4e24c33765c08b7691810e68d879365d"
+checksum = "062879d46aa4c9dfe0d33b035bbaf512da192131645d05deacb7033ec8581a09"
dependencies = [
"heck 0.5.0",
"log",
@@ -6759,9 +6785,9 @@ dependencies = [
[[package]]
name = "typify-macro"
-version = "0.4.2"
+version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7560adf816a1e8dad7c63d8845ef6e31e673e39eab310d225636779230cbedeb"
+checksum = "9708a3ceb6660ba3f8d2b8f0567e7d4b8b198e2b94d093b8a6077a751425de9e"
dependencies = [
"proc-macro2",
"quote",
@@ -6782,9 +6808,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5"
[[package]]
name = "unicode-ident"
-version = "1.0.18"
+version = "1.0.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5a5f39404a5da50712a4c1eecf25e90dd62b613502b7e925fd4e4d19b5c96512"
+checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d"
[[package]]
name = "unicode-normalization"
@@ -6930,11 +6956,20 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
[[package]]
name = "wasi"
-version = "0.14.2+wasi-0.2.4"
+version = "0.14.7+wasi-0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "883478de20367e224c0090af9cf5f9fa85bed63a95c1abf3afc5c083ebc06e8c"
+dependencies = [
+ "wasip2",
+]
+
+[[package]]
+name = "wasip2"
+version = "1.0.1+wasi-0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9683f9a5a998d873c0d21fcbe3c083009670149a8fab228644b8bd36b2c48cb3"
+checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7"
dependencies = [
- "wit-bindgen-rt",
+ "wit-bindgen",
]
[[package]]
@@ -6945,9 +6980,9 @@ checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b"
[[package]]
name = "wasm-bindgen"
-version = "0.2.101"
+version = "0.2.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e14915cadd45b529bb8d1f343c4ed0ac1de926144b746e2710f9cd05df6603b"
+checksum = "c1da10c01ae9f1ae40cbfac0bac3b1e724b320abfcf52229f80b547c0d250e2d"
dependencies = [
"cfg-if",
"once_cell",
@@ -6958,9 +6993,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-backend"
-version = "0.2.101"
+version = "0.2.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e28d1ba982ca7923fd01448d5c30c6864d0a14109560296a162f80f305fb93bb"
+checksum = "671c9a5a66f49d8a47345ab942e2cb93c7d1d0339065d4f8139c486121b43b19"
dependencies = [
"bumpalo",
"log",
@@ -6972,9 +7007,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-futures"
-version = "0.4.51"
+version = "0.4.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0ca85039a9b469b38336411d6d6ced91f3fc87109a2a27b0c197663f5144dffe"
+checksum = "7e038d41e478cc73bae0ff9b36c60cff1c98b8f38f8d7e8061e79ee63608ac5c"
dependencies = [
"cfg-if",
"js-sys",
@@ -6985,9 +7020,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro"
-version = "0.2.101"
+version = "0.2.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7c3d463ae3eff775b0c45df9da45d68837702ac35af998361e2c84e7c5ec1b0d"
+checksum = "7ca60477e4c59f5f2986c50191cd972e3a50d8a95603bc9434501cf156a9a119"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
@@ -6995,9 +7030,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro-support"
-version = "0.2.101"
+version = "0.2.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bb4ce89b08211f923caf51d527662b75bdc9c9c7aab40f86dcb9fb85ac552aa"
+checksum = "9f07d2f20d4da7b26400c9f4a0511e6e0345b040694e8a75bd41d578fa4421d7"
dependencies = [
"proc-macro2",
"quote",
@@ -7008,18 +7043,18 @@ dependencies = [
[[package]]
name = "wasm-bindgen-shared"
-version = "0.2.101"
+version = "0.2.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f143854a3b13752c6950862c906306adb27c7e839f7414cec8fea35beab624c1"
+checksum = "bad67dc8b2a1a6e5448428adec4c3e84c43e561d8c9ee8a9e5aabeb193ec41d1"
dependencies = [
"unicode-ident",
]
[[package]]
name = "wasm-bindgen-test"
-version = "0.3.51"
+version = "0.3.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "80cc7f8a4114fdaa0c58383caf973fc126cf004eba25c9dc639bccd3880d55ad"
+checksum = "4e381134e148c1062f965a42ed1f5ee933eef2927c3f70d1812158f711d39865"
dependencies = [
"js-sys",
"minicov",
@@ -7030,9 +7065,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-test-macro"
-version = "0.3.51"
+version = "0.3.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c5ada2ab788d46d4bda04c9d567702a79c8ced14f51f221646a16ed39d0e6a5d"
+checksum = "b673bca3298fe582aeef8352330ecbad91849f85090805582400850f8270a2e8"
dependencies = [
"proc-macro2",
"quote",
@@ -7054,9 +7089,9 @@ dependencies = [
[[package]]
name = "web-sys"
-version = "0.3.78"
+version = "0.3.81"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "77e4b637749ff0d92b8fad63aa1f7cff3cbe125fd49c175cd6345e7272638b12"
+checksum = "9367c417a924a74cae129e6a2ae3b47fabb1f8995595ab474029da749a8be120"
dependencies = [
"js-sys",
"wasm-bindgen",
@@ -7072,18 +7107,6 @@ dependencies = [
"wasm-bindgen",
]
-[[package]]
-name = "which"
-version = "4.4.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "87ba24419a2078cd2b0f2ede2691b6c66d8e47836da3b6db8265ebad47afbfc7"
-dependencies = [
- "either",
- "home",
- "once_cell",
- "rustix 0.38.44",
-]
-
[[package]]
name = "whoami"
version = "1.6.1"
@@ -7113,11 +7136,11 @@ checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-util"
-version = "0.1.9"
+version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cf221c93e13a30d793f7645a0e7762c55d169dbb0a49671918a2319d289b10bb"
+checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
- "windows-sys 0.59.0",
+ "windows-sys 0.61.0",
]
[[package]]
@@ -7133,7 +7156,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9babd3a767a4c1aef6900409f85f5d53ce2544ccdfaa86dad48c91782c6d6893"
dependencies = [
"windows-collections",
- "windows-core",
+ "windows-core 0.61.2",
"windows-future",
"windows-link 0.1.3",
"windows-numerics",
@@ -7145,7 +7168,7 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3beeceb5e5cfd9eb1d76b381630e82c4241ccd0d27f1a39ed41b2760b255c5e8"
dependencies = [
- "windows-core",
+ "windows-core 0.61.2",
]
[[package]]
@@ -7157,8 +7180,21 @@ dependencies = [
"windows-implement",
"windows-interface",
"windows-link 0.1.3",
- "windows-result",
- "windows-strings",
+ "windows-result 0.3.4",
+ "windows-strings 0.4.2",
+]
+
+[[package]]
+name = "windows-core"
+version = "0.62.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "57fe7168f7de578d2d8a05b07fd61870d2e73b4020e9f49aa00da8471723497c"
+dependencies = [
+ "windows-implement",
+ "windows-interface",
+ "windows-link 0.2.0",
+ "windows-result 0.4.0",
+ "windows-strings 0.5.0",
]
[[package]]
@@ -7167,7 +7203,7 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc6a41e98427b19fe4b73c550f060b59fa592d7d686537eebf9385621bfbad8e"
dependencies = [
- "windows-core",
+ "windows-core 0.61.2",
"windows-link 0.1.3",
"windows-threading",
]
@@ -7212,7 +7248,7 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9150af68066c4c5c07ddc0ce30421554771e528bde427614c61038bc2c92c2b1"
dependencies = [
- "windows-core",
+ "windows-core 0.61.2",
"windows-link 0.1.3",
]
@@ -7225,6 +7261,15 @@ dependencies = [
"windows-link 0.1.3",
]
+[[package]]
+name = "windows-result"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7084dcc306f89883455a206237404d3eaf961e5bd7e0f312f7c91f57eb44167f"
+dependencies = [
+ "windows-link 0.2.0",
+]
+
[[package]]
name = "windows-strings"
version = "0.4.2"
@@ -7234,6 +7279,15 @@ dependencies = [
"windows-link 0.1.3",
]
+[[package]]
+name = "windows-strings"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7218c655a553b0bed4426cf54b20d7ba363ef543b52d515b3e48d7fd55318dda"
+dependencies = [
+ "windows-link 0.2.0",
+]
+
[[package]]
name = "windows-sys"
version = "0.52.0"
@@ -7261,6 +7315,15 @@ dependencies = [
"windows-targets 0.53.3",
]
+[[package]]
+name = "windows-sys"
+version = "0.61.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e201184e40b2ede64bc2ea34968b28e33622acdbbf37104f0e4a33f7abe657aa"
+dependencies = [
+ "windows-link 0.2.0",
+]
+
[[package]]
name = "windows-targets"
version = "0.52.6"
@@ -7401,21 +7464,18 @@ checksum = "271414315aff87387382ec3d271b52d7ae78726f5d44ac98b4f4030c91880486"
[[package]]
name = "winnow"
-version = "0.7.12"
+version = "0.7.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f3edebf492c8125044983378ecb5766203ad3b4c2f7a922bd7dd207f6d443e95"
+checksum = "21a0236b59786fed61e2a80582dd500fe61f18b5dca67a4a067d0bc9039339cf"
dependencies = [
"memchr",
]
[[package]]
-name = "wit-bindgen-rt"
-version = "0.39.0"
+name = "wit-bindgen"
+version = "0.46.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6f42320e61fe2cfd34354ecb597f86f413484a798ba44a8ca1165c58d42da6c1"
-dependencies = [
- "bitflags 2.9.1",
-]
+checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59"
[[package]]
name = "writeable"
@@ -7434,12 +7494,12 @@ dependencies = [
[[package]]
name = "xattr"
-version = "1.5.1"
+version = "1.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "af3a19837351dc82ba89f8a125e22a3c475f05aba604acc023d62b2739ae2909"
+checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156"
dependencies = [
"libc",
- "rustix 1.0.8",
+ "rustix",
]
[[package]]
@@ -7489,18 +7549,18 @@ dependencies = [
[[package]]
name = "zerocopy"
-version = "0.8.26"
+version = "0.8.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1039dd0d3c310cf05de012d8a39ff557cb0d23087fd44cad61df08fc31907a2f"
+checksum = "0894878a5fa3edfd6da3f88c4805f4c8558e2b996227a3d864f47fe11e38282c"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
-version = "0.8.26"
+version = "0.8.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9ecf5b4cc5364572d7f4c329661bcc82724222973f2cab6f050a4e5c22f75181"
+checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831"
dependencies = [
"proc-macro2",
"quote",
@@ -7569,9 +7629,9 @@ dependencies = [
[[package]]
name = "zlib-rs"
-version = "0.5.1"
+version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "626bd9fa9734751fc50d6060752170984d7053f5a39061f524cda68023d4db8a"
+checksum = "2f06ae92f42f5e5c42443fd094f245eb656abf56dd7cce9b8b263236565e00f2"
[[package]]
name = "zstd"
@@ -7593,9 +7653,9 @@ dependencies = [
[[package]]
name = "zstd-sys"
-version = "2.0.15+zstd.1.5.7"
+version = "2.0.16+zstd.1.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eb81183ddd97d0c74cedf1d50d85c8d08c1b8b68ee863bdee9e706eedba1a237"
+checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748"
dependencies = [
"cc",
"pkg-config",
diff --git a/Cargo.toml b/Cargo.toml
index 92392a199..b54a75e5d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -76,7 +76,7 @@ license = "Apache-2.0"
readme = "README.md"
repository = "https://github.com/apache/datafusion"
# Define Minimum Supported Rust Version (MSRV)
-rust-version = "1.86.0"
+rust-version = "1.87.0"
# Define DataFusion version
version = "50.0.0"
@@ -90,19 +90,19 @@ ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
apache-avro = { version = "0.20", default-features = false }
-arrow = { version = "56.0.0", features = [
+arrow = { version = "56.2.0", features = [
"prettyprint",
"chrono-tz",
] }
-arrow-buffer = { version = "56.0.0", default-features = false }
-arrow-flight = { version = "56.0.0", features = [
+arrow-buffer = { version = "56.2.0", default-features = false }
+arrow-flight = { version = "56.2.0", features = [
"flight-sql-experimental",
] }
-arrow-ipc = { version = "56.0.0", default-features = false, features = [
+arrow-ipc = { version = "56.2.0", default-features = false, features = [
"lz4",
] }
-arrow-ord = { version = "56.0.0", default-features = false }
-arrow-schema = { version = "56.0.0", default-features = false }
+arrow-ord = { version = "56.2.0", default-features = false }
+arrow-schema = { version = "56.2.0", default-features = false }
async-trait = "0.1.89"
bigdecimal = "0.4.8"
bytes = "1.10"
@@ -152,16 +152,15 @@ futures = "0.3"
half = { version = "2.6.0", default-features = false }
hashbrown = { version = "0.14.5", features = ["raw"] }
hex = { version = "0.4.3" }
-indexmap = "2.11.3"
+indexmap = "2.11.4"
itertools = "0.14"
log = "^0.4"
-object_store = { version = "0.12.3", default-features = false }
+object_store = { version = "0.12.4", default-features = false }
parking_lot = "0.12"
-parquet = { version = "56.0.0", default-features = false, features = [
+parquet = { version = "56.2.0", default-features = false, features = [
"arrow",
"async",
"object_store",
- "encryption",
] }
pbjson = { version = "0.7.0" }
pbjson-types = "0.7"
@@ -180,13 +179,56 @@ testcontainers-modules = { version = "0.12" }
tokio = { version = "1.47", features = ["macros", "rt", "sync"] }
url = "2.5.7"
+[workspace.lints.clippy]
+# Detects large stack-allocated futures that may cause stack overflow crashes (see threshold in clippy.toml)
+large_futures = "warn"
+used_underscore_binding = "warn"
+or_fun_call = "warn"
+unnecessary_lazy_evaluations = "warn"
+uninlined_format_args = "warn"
+inefficient_to_string = "warn"
+
+[workspace.lints.rust]
+unexpected_cfgs = { level = "warn", check-cfg = [
+ 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))',
+ "cfg(tarpaulin)",
+ "cfg(tarpaulin_include)",
+] }
+unused_qualifications = "deny"
+
+# --------------------
+# Compilation Profiles
+# --------------------
+# A Cargo profile is a preset for the compiler/linker knobs that trade off:
+# - Build time: how quickly code compiles and links
+# - Runtime performance: how fast the resulting binaries execute
+# - Binary size: how large the executables end up
+# - Debuggability: how much debug information is preserved for debugging and profiling
+#
+# Profiles available:
+# - dev: default debug build; fastest to compile, slowest to run, full debug info
+# for everyday development.
+# Run: cargo run
+# - release: optimized build; slowest to compile, fastest to run, smallest
+# binaries for public releases.
+# Run: cargo run --release
+# - release-nonlto: skips LTO, so it builds quicker while staying close to
+# release performance. It is useful when developing performance optimizations.
+# Run: cargo run --profile release-nonlto
+# - profiling: inherits release optimizations but retains debug info to support
+# profiling tools and flamegraphs.
+# Run: cargo run --profile profiling
+# - ci: derived from `dev` but disables incremental builds and strips dependency
+# symbols to keep CI artifacts small and reproducible.
+# Run: cargo run --profile ci
+#
+# If you want to optimize compilation, the `compile_profile` benchmark can be useful.
+# See `benchmarks/README.md` for more details.
[profile.release]
codegen-units = 1
lto = true
strip = true # Eliminate debug information to minimize binary size
-# the release profile takes a long time to build so we can use this profile during development to save time
-# cargo build --profile release-nonlto
[profile.release-nonlto]
codegen-units = 16
debug-assertions = false
@@ -199,36 +241,20 @@ rpath = false
strip = false # Retain debug info for flamegraphs
[profile.ci]
+debug = false
inherits = "dev"
incremental = false
-# ci turns off debug info, etc. for dependencies to allow for smaller binaries making caching more effective
+# This rule applies to every package except workspace members (dependencies
+# such as `arrow` and `tokio`). It disables debug info and related features on
+# dependencies so their binaries stay smaller, improving cache reuse.
[profile.ci.package."*"]
debug = false
debug-assertions = false
strip = "debuginfo"
incremental = false
-# release inherited profile keeping debug information and symbols
-# for mem/cpu profiling
[profile.profiling]
inherits = "release"
debug = true
strip = false
-
-[workspace.lints.clippy]
-# Detects large stack-allocated futures that may cause stack overflow crashes (see threshold in clippy.toml)
-large_futures = "warn"
-used_underscore_binding = "warn"
-or_fun_call = "warn"
-unnecessary_lazy_evaluations = "warn"
-uninlined_format_args = "warn"
-inefficient_to_string = "warn"
-
-[workspace.lints.rust]
-unexpected_cfgs = { level = "warn", check-cfg = [
- 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))',
- "cfg(tarpaulin)",
- "cfg(tarpaulin_include)",
-] }
-unused_qualifications = "deny"
diff --git a/benchmarks/Cargo.toml b/benchmarks/Cargo.toml
index 2225d9982..b3fd52081 100644
--- a/benchmarks/Cargo.toml
+++ b/benchmarks/Cargo.toml
@@ -48,7 +48,7 @@ object_store = { workspace = true }
parquet = { workspace = true, default-features = true }
rand = { workspace = true }
regex.workspace = true
-serde = { version = "1.0.219", features = ["derive"] }
+serde = { version = "1.0.228", features = ["derive"] }
serde_json = { workspace = true }
snmalloc-rs = { version = "0.3", optional = true }
structopt = { version = "0.3", default-features = false }
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 872500ef8..8fed85fa0 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -87,6 +87,39 @@ To run for specific query, for example Q21
./bench.sh run tpch10 21
```
+## Compile profile benchmark
+
+Generate the data required for the compile profile helper (TPC-H SF=1):
+
+```shell
+./bench.sh data compile_profile
+```
+
+Run the benchmark across all default Cargo profiles (`dev`, `release`, `ci`, `release-nonlto`):
+
+```shell
+./bench.sh run compile_profile
+```
+
+Limit the run to a single profile:
+
+```shell
+./bench.sh run compile_profile dev
+```
+
+Or specify a subset of profiles:
+
+```shell
+./bench.sh run compile_profile dev release
+```
+
+You can also invoke the helper directly if you need to customise arguments further:
+
+```shell
+./benchmarks/compile_profile.py --profiles dev release --data /path/to/tpch_sf1
+```
+
+
## Benchmark with modified configurations
### Select join algorithm
@@ -727,6 +760,20 @@ Different queries are included to test nested loop joins under various workloads
./bench.sh run nlj
```
+## Hash Join
+
+This benchmark focuses on the performance of queries with nested hash joins, minimizing other overheads such as scanning data sources or evaluating predicates.
+
+Several queries are included to test hash joins under various workloads.
+
+### Example Run
+
+```bash
+# No need to generate data: this benchmark uses table function `range()` as the data source
+
+./bench.sh run hj
+```
+
## Cancellation
Test performance of cancelling queries.
diff --git a/benchmarks/bench.sh b/benchmarks/bench.sh
index b99ab0100..dbfd319dd 100755
--- a/benchmarks/bench.sh
+++ b/benchmarks/bench.sh
@@ -125,6 +125,9 @@ imdb: Join Order Benchmark (JOB) using the IMDB dataset conver
# Micro-Benchmarks (specific operators and features)
cancellation: How long cancelling a query takes
nlj: Benchmark for simple nested loop joins, testing various join scenarios
+hj: Benchmark for simple hash joins, testing various join scenarios
+compile_profile: Compile and execute TPC-H across selected Cargo profiles, reporting timing and binary size
+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Supported Configuration (Environment Variables)
@@ -304,6 +307,13 @@ main() {
# nlj uses range() function, no data generation needed
echo "NLJ benchmark does not require data generation"
;;
+ hj)
+ # hj uses range() function, no data generation needed
+ echo "HJ benchmark does not require data generation"
+ ;;
+ compile_profile)
+ data_tpch "1"
+ ;;
*)
echo "Error: unknown benchmark '$BENCHMARK' for data generation"
usage
@@ -313,20 +323,32 @@ main() {
run)
# Parse positional parameters
BENCHMARK=${ARG2:-"${BENCHMARK}"}
+ EXTRA_ARGS=("${POSITIONAL_ARGS[@]:2}")
+ PROFILE_ARGS=()
+ QUERY=""
+ QUERY_ARG=""
+ if [ "$BENCHMARK" = "compile_profile" ]; then
+ PROFILE_ARGS=("${EXTRA_ARGS[@]}")
+ else
+ QUERY=${EXTRA_ARGS[0]}
+ if [ -n "$QUERY" ]; then
+ QUERY_ARG="--query ${QUERY}"
+ fi
+ fi
BRANCH_NAME=$(cd "${DATAFUSION_DIR}" && git rev-parse --abbrev-ref HEAD)
BRANCH_NAME=${BRANCH_NAME//\//_} # mind blowing syntax to replace / with _
RESULTS_NAME=${RESULTS_NAME:-"${BRANCH_NAME}"}
RESULTS_DIR=${RESULTS_DIR:-"$SCRIPT_DIR/results/$RESULTS_NAME"}
- # Optional query filter to run specific query
- QUERY=${ARG3}
- QUERY_ARG=$([ -n "$QUERY" ] && echo "--query ${QUERY}" || echo "")
-
echo "***************************"
echo "DataFusion Benchmark Script"
echo "COMMAND: ${COMMAND}"
echo "BENCHMARK: ${BENCHMARK}"
- echo "QUERY: ${QUERY:-All}"
+ if [ "$BENCHMARK" = "compile_profile" ]; then
+ echo "PROFILES: ${PROFILE_ARGS[*]:-All}"
+ else
+ echo "QUERY: ${QUERY:-All}"
+ fi
echo "DATAFUSION_DIR: ${DATAFUSION_DIR}"
echo "BRANCH_NAME: ${BRANCH_NAME}"
echo "DATA_DIR: ${DATA_DIR}"
@@ -361,6 +383,7 @@ main() {
run_imdb
run_external_aggr
run_nlj
+ run_hj
;;
tpch)
run_tpch "1" "parquet"
@@ -468,6 +491,12 @@ main() {
nlj)
run_nlj
;;
+ hj)
+ run_hj
+ ;;
+ compile_profile)
+ run_compile_profile "${PROFILE_ARGS[@]}"
+ ;;
*)
echo "Error: unknown benchmark '$BENCHMARK' for run"
usage
@@ -593,6 +622,20 @@ run_tpch_mem() {
debug_run $CARGO_COMMAND --bin tpch -- benchmark datafusion --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" -m --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG}
}
+# Runs the compile profile benchmark helper
+run_compile_profile() {
+ local profiles=("$@")
+ local runner="${SCRIPT_DIR}/compile_profile.py"
+ local data_path="${DATA_DIR}/tpch_sf1"
+
+ echo "Running compile profile benchmark..."
+ local cmd=(python3 "${runner}" --data "${data_path}")
+ if [ ${#profiles[@]} -gt 0 ]; then
+ cmd+=(--profiles "${profiles[@]}")
+ fi
+ debug_run "${cmd[@]}"
+}
+
# Runs the cancellation benchmark
run_cancellation() {
RESULTS_FILE="${RESULTS_DIR}/cancellation.json"
@@ -1103,6 +1146,14 @@ run_nlj() {
debug_run $CARGO_COMMAND --bin dfbench -- nlj --iterations 5 -o "${RESULTS_FILE}" ${QUERY_ARG}
}
+# Runs the hj benchmark
+run_hj() {
+ RESULTS_FILE="${RESULTS_DIR}/hj.json"
+ echo "RESULTS_FILE: ${RESULTS_FILE}"
+ echo "Running hj benchmark..."
+ debug_run $CARGO_COMMAND --bin dfbench -- hj --iterations 5 -o "${RESULTS_FILE}" ${QUERY_ARG}
+}
+
compare_benchmarks() {
BASE_RESULTS_DIR="${SCRIPT_DIR}/results"
diff --git a/benchmarks/compile_profile.py b/benchmarks/compile_profile.py
new file mode 100644
index 000000000..ae51de949
--- /dev/null
+++ b/benchmarks/compile_profile.py
@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Compile profile benchmark runner for DataFusion.
+
+Builds the `tpch` benchmark binary with several Cargo profiles (e.g. `--release` or `--profile ci`), runs the full TPC-H suite against the Parquet data under `benchmarks/data/tpch_sf1`, and reports compile time, execution time, and resulting
+binary size.
+
+See `benchmarks/README.md` for usages.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Iterable, NamedTuple
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_DATA_DIR = REPO_ROOT / "benchmarks" / "data" / "tpch_sf1"
+DEFAULT_ITERATIONS = 1
+DEFAULT_FORMAT = "parquet"
+DEFAULT_PARTITIONS: int | None = None
+TPCH_BINARY = "tpch.exe" if os.name == "nt" else "tpch"
+PROFILE_TARGET_DIR = {
+ "dev": "debug",
+ "release": "release",
+ "ci": "ci",
+ "release-nonlto": "release-nonlto",
+}
+
+
+class ProfileResult(NamedTuple):
+ profile: str
+ compile_seconds: float
+ run_seconds: float
+ binary_bytes: int
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--profiles",
+ nargs="+",
+ default=list(PROFILE_TARGET_DIR.keys()),
+ help="Cargo profiles to test (default: dev release ci release-nonlto)",
+ )
+ parser.add_argument(
+ "--data",
+ type=Path,
+ default=DEFAULT_DATA_DIR,
+ help="Path to TPCH dataset (default: benchmarks/data/tpch_sf1)",
+ )
+ return parser.parse_args()
+
+
+def timed_run(command: Iterable[str]) -> float:
+ start = time.perf_counter()
+ try:
+ subprocess.run(command, cwd=REPO_ROOT, check=True)
+ except subprocess.CalledProcessError as exc:
+ raise RuntimeError(f"command failed: {' '.join(command)}") from exc
+ return time.perf_counter() - start
+
+
+def cargo_build(profile: str) -> float:
+ if profile == "dev":
+ command = ["cargo", "build", "--bin", "tpch"]
+ else:
+ command = ["cargo", "build", "--profile", profile, "--bin", "tpch"]
+ return timed_run(command)
+
+
+def cargo_clean(profile: str) -> None:
+ command = ["cargo", "clean", "--profile", profile]
+ try:
+ subprocess.run(command, cwd=REPO_ROOT, check=True)
+ except subprocess.CalledProcessError as exc:
+ raise RuntimeError(f"failed to clean cargo artifacts for profile '{profile}'") from exc
+
+
+def run_benchmark(profile: str, data_path: Path) -> float:
+ binary_dir = PROFILE_TARGET_DIR.get(profile)
+ if not binary_dir:
+ raise ValueError(f"unknown profile '{profile}'")
+ binary_path = REPO_ROOT / "target" / binary_dir / TPCH_BINARY
+ if not binary_path.exists():
+ raise FileNotFoundError(f"compiled binary not found at {binary_path}")
+
+ command = [
+ str(binary_path),
+ "benchmark",
+ "datafusion",
+ "--iterations",
+ str(DEFAULT_ITERATIONS),
+ "--path",
+ str(data_path),
+ "--format",
+ DEFAULT_FORMAT,
+ ]
+ if DEFAULT_PARTITIONS is not None:
+ command.extend(["--partitions", str(DEFAULT_PARTITIONS)])
+ env = os.environ.copy()
+ env.setdefault("RUST_LOG", "warn")
+
+ start = time.perf_counter()
+ try:
+ subprocess.run(command, cwd=REPO_ROOT, env=env, check=True)
+ except subprocess.CalledProcessError as exc:
+ raise RuntimeError(f"benchmark failed for profile '{profile}'") from exc
+ return time.perf_counter() - start
+
+
+def binary_size(profile: str) -> int:
+ binary_dir = PROFILE_TARGET_DIR[profile]
+ binary_path = REPO_ROOT / "target" / binary_dir / TPCH_BINARY
+ return binary_path.stat().st_size
+
+
+def human_time(seconds: float) -> str:
+ return f"{seconds:6.2f}s"
+
+
+def human_size(size: int) -> str:
+ value = float(size)
+ for unit in ("B", "KB", "MB", "GB", "TB"):
+ if value < 1024 or unit == "TB":
+ return f"{value:6.1f}{unit}"
+ value /= 1024
+ return f"{value:6.1f}TB"
+
+
+def main() -> None:
+ args = parse_args()
+ data_path = args.data.resolve()
+ if not data_path.exists():
+ print(f"Data directory not found: {data_path}", file=sys.stderr)
+ sys.exit(1)
+
+ results: list[ProfileResult] = []
+ for profile in args.profiles:
+ print(f"\n=== Profile: {profile} ===")
+ print("Cleaning previous build artifacts...")
+ cargo_clean(profile)
+ compile_seconds = cargo_build(profile)
+ run_seconds = run_benchmark(profile, data_path)
+ size_bytes = binary_size(profile)
+ results.append(ProfileResult(profile, compile_seconds, run_seconds, size_bytes))
+
+ print("\nSummary")
+ header = f"{'Profile':<15}{'Compile':>12}{'Run':>12}{'Size':>12}"
+ print(header)
+ print("-" * len(header))
+ for result in results:
+ print(
+ f"{result.profile:<15}{human_time(result.compile_seconds):>12}"
+ f"{human_time(result.run_seconds):>12}{human_size(result.binary_bytes):>12}"
+ )
+
+if __name__ == "__main__":
+ main()
diff --git a/benchmarks/src/bin/dfbench.rs b/benchmarks/src/bin/dfbench.rs
index 88378492b..816cae0e3 100644
--- a/benchmarks/src/bin/dfbench.rs
+++ b/benchmarks/src/bin/dfbench.rs
@@ -33,7 +33,9 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;
#[global_allocator]
static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
-use datafusion_benchmarks::{cancellation, clickbench, h2o, imdb, nlj, sort_tpch, tpch};
+use datafusion_benchmarks::{
+ cancellation, clickbench, h2o, hj, imdb, nlj, sort_tpch, tpch,
+};
#[derive(Debug, StructOpt)]
#[structopt(about = "benchmark command")]
@@ -41,6 +43,7 @@ enum Options {
Cancellation(cancellation::RunOpt),
Clickbench(clickbench::RunOpt),
H2o(h2o::RunOpt),
+ HJ(hj::RunOpt),
Imdb(imdb::RunOpt),
Nlj(nlj::RunOpt),
SortTpch(sort_tpch::RunOpt),
@@ -57,6 +60,7 @@ pub async fn main() -> Result<()> {
Options::Cancellation(opt) => opt.run().await,
Options::Clickbench(opt) => opt.run().await,
Options::H2o(opt) => opt.run().await,
+ Options::HJ(opt) => opt.run().await,
Options::Imdb(opt) => Box::pin(opt.run()).await,
Options::Nlj(opt) => opt.run().await,
Options::SortTpch(opt) => opt.run().await,
diff --git a/benchmarks/src/hj.rs b/benchmarks/src/hj.rs
new file mode 100644
index 000000000..505b32274
--- /dev/null
+++ b/benchmarks/src/hj.rs
@@ -0,0 +1,273 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::util::{BenchmarkRun, CommonOpt, QueryResult};
+use datafusion::physical_plan::execute_stream;
+use datafusion::{error::Result, prelude::SessionContext};
+use datafusion_common::instant::Instant;
+use datafusion_common::{exec_datafusion_err, exec_err, DataFusionError};
+use structopt::StructOpt;
+
+use futures::StreamExt;
+
+// TODO: Add existence joins
+
+/// Run the Hash Join benchmark
+///
+/// This micro-benchmark focuses on the performance characteristics of Hash Joins.
+/// It uses simple equality predicates to ensure a hash join is selected.
+/// Where we vary selectivity, we do so with additional cheap predicates that
+/// do not change the join key (so the physical operator remains HashJoin).
+#[derive(Debug, StructOpt, Clone)]
+#[structopt(verbatim_doc_comment)]
+pub struct RunOpt {
+ /// Query number (between 1 and 12). If not specified, runs all queries
+ #[structopt(short, long)]
+ query: Option<usize>,
+
+ /// Common options (iterations, batch size, target_partitions, etc.)
+ #[structopt(flatten)]
+ common: CommonOpt,
+
+ /// If present, write results json here
+ #[structopt(parse(from_os_str), short = "o", long = "output")]
+ output_path: Option<std::path::PathBuf>,
+}
+
+/// Inline SQL queries for Hash Join benchmarks
+///
+/// Each query's comment includes:
+/// - Left row count × Right row count
+/// - Join predicate selectivity (approximate output fraction).
+/// - Q11 and Q12 selectivity is relative to cartesian product while the others are
+/// relative to probe side.
+const HASH_QUERIES: &[&str] = &[
+ // Q1: INNER 10 x 10K | LOW ~0.1%
+ // equality on key + cheap filter to downselect
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 9000, 1000) AS t1(value)
+ JOIN range(10000) AS t2
+ ON t1.value = t2.value;
+ "#,
+ // Q2: INNER 10 x 10K | LOW ~0.1%
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 9000, 1000) AS t1
+ JOIN range(10000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 5 = 0
+ "#,
+ // Q3: INNER 10K x 10K | HIGH ~90%
+ r#"
+ SELECT t1.value, t2.value
+ FROM range(10000) AS t1
+ JOIN range(10000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 10 <> 0
+ "#,
+ // Q4: INNER 30 x 30K | LOW ~0.1%
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 29000, 1000) AS t1
+ JOIN range(30000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 5 = 0
+ "#,
+ // Q5: INNER 10 x 200K | VERY LOW ~0.005% (small to large)
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 9000, 1000) AS t1
+ JOIN range(200000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 1000 = 0
+ "#,
+ // Q6: INNER 200K x 10 | VERY LOW ~0.005% (large to small)
+ r#"
+ SELECT t1.value, t2.value
+ FROM range(200000) AS t1
+ JOIN generate_series(0, 9000, 1000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 1000 = 0
+ "#,
+ // Q7: RIGHT OUTER 10 x 200K | LOW ~0.1%
+ // Outer join still uses HashJoin for equi-keys; the extra filter reduces matches
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM generate_series(0, 9000, 1000) AS t1
+ RIGHT JOIN range(200000) AS t2
+ ON t1.value = t2.value
+ WHERE t2.value % 1000 = 0
+ "#,
+ // Q8: LEFT OUTER 200K x 10 | LOW ~0.1%
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM range(200000) AS t1
+ LEFT JOIN generate_series(0, 9000, 1000) AS t2
+ ON t1.value = t2.value
+ WHERE t1.value % 1000 = 0
+ "#,
+ // Q9: FULL OUTER 30 x 30K | LOW ~0.1%
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM generate_series(0, 29000, 1000) AS t1
+ FULL JOIN range(30000) AS t2
+ ON t1.value = t2.value
+ WHERE COALESCE(t1.value, t2.value) % 1000 = 0
+ "#,
+ // Q10: FULL OUTER 30 x 30K | HIGH ~90%
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM generate_series(0, 29000, 1000) AS t1
+ FULL JOIN range(30000) AS t2
+ ON t1.value = t2.value
+ WHERE COALESCE(t1.value, t2.value) % 10 <> 0
+ "#,
+ // Q11: INNER 30 x 30K | MEDIUM ~50% | cheap predicate on parity
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 29000, 1000) AS t1
+ INNER JOIN range(30000) AS t2
+ ON (t1.value % 2) = (t2.value % 2)
+ "#,
+ // Q12: FULL OUTER 30 x 30K | MEDIUM ~50% | expression key
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM generate_series(0, 29000, 1000) AS t1
+ FULL JOIN range(30000) AS t2
+ ON (t1.value % 2) = (t2.value % 2)
+ "#,
+ // Q13: INNER 30 x 30K | LOW 0.1% | modulo with adding values
+ r#"
+ SELECT t1.value, t2.value
+ FROM generate_series(0, 29000, 1000) AS t1
+ INNER JOIN range(30000) AS t2
+ ON (t1.value = t2.value) AND ((t1.value + t2.value) % 10 < 1)
+ "#,
+ // Q14: FULL OUTER 30 x 30K | ALL ~100% | modulo
+ r#"
+ SELECT t1.value AS l, t2.value AS r
+ FROM generate_series(0, 29000, 1000) AS t1
+ FULL JOIN range(30000) AS t2
+ ON (t1.value = t2.value) AND ((t1.value + t2.value) % 10 = 0)
+ "#,
+];
+
+impl RunOpt {
+ pub async fn run(self) -> Result<()> {
+ println!("Running Hash Join benchmarks with the following options: {self:#?}\n");
+
+ let query_range = match self.query {
+ Some(query_id) => {
+ if query_id >= 1 && query_id <= HASH_QUERIES.len() {
+ query_id..=query_id
+ } else {
+ return exec_err!(
+ "Query {query_id} not found. Available queries: 1 to {}",
+ HASH_QUERIES.len()
+ );
+ }
+ }
+ None => 1..=HASH_QUERIES.len(),
+ };
+
+ let config = self.common.config()?;
+ let rt_builder = self.common.runtime_env_builder()?;
+ let ctx = SessionContext::new_with_config_rt(config, rt_builder.build_arc()?);
+
+ let mut benchmark_run = BenchmarkRun::new();
+
+ for query_id in query_range {
+ let query_index = query_id - 1;
+ let sql = HASH_QUERIES[query_index];
+
+ benchmark_run.start_new_case(&format!("Query {query_id}"));
+ let query_run = self.benchmark_query(sql, &query_id.to_string(), &ctx).await;
+ match query_run {
+ Ok(query_results) => {
+ for iter in query_results {
+ benchmark_run.write_iter(iter.elapsed, iter.row_count);
+ }
+ }
+ Err(e) => {
+ return Err(DataFusionError::Context(
+ format!("Hash Join benchmark Q{query_id} failed with error:"),
+ Box::new(e),
+ ));
+ }
+ }
+ }
+
+ benchmark_run.maybe_write_json(self.output_path.as_ref())?;
+ Ok(())
+ }
+
+ /// Validates that the physical plan uses a HashJoin, then executes.
+ async fn benchmark_query(
+ &self,
+ sql: &str,
+ query_name: &str,
+ ctx: &SessionContext,
+ ) -> Result<Vec<QueryResult>> {
+ let mut query_results = vec![];
+
+ // Build/validate plan
+ let df = ctx.sql(sql).await?;
+ let physical_plan = df.create_physical_plan().await?;
+ let plan_string = format!("{physical_plan:#?}");
+
+ if !plan_string.contains("HashJoinExec") {
+ return Err(exec_datafusion_err!(
+ "Query {query_name} does not use Hash Join. Physical plan: {plan_string}"
+ ));
+ }
+
+ // Execute without buffering
+ for i in 0..self.common.iterations {
+ let start = Instant::now();
+ let row_count = Self::execute_sql_without_result_buffering(sql, ctx).await?;
+ let elapsed = start.elapsed();
+
+ println!(
+ "Query {query_name} iteration {i} returned {row_count} rows in {elapsed:?}"
+ );
+
+ query_results.push(QueryResult { elapsed, row_count });
+ }
+
+ Ok(query_results)
+ }
+
+ /// Executes the SQL query and drops each batch to avoid result buffering.
+ async fn execute_sql_without_result_buffering(
+ sql: &str,
+ ctx: &SessionContext,
+ ) -> Result<usize> {
+ let mut row_count = 0;
+
+ let df = ctx.sql(sql).await?;
+ let physical_plan = df.create_physical_plan().await?;
+ let mut stream = execute_stream(physical_plan, ctx.task_ctx())?;
+
+ while let Some(batch) = stream.next().await {
+ row_count += batch?.num_rows();
+ // Drop batches immediately to minimize memory pressure
+ }
+
+ Ok(row_count)
+ }
+}
diff --git a/benchmarks/src/imdb/run.rs b/benchmarks/src/imdb/run.rs
index 90e0947f6..3d58d5f54 100644
--- a/benchmarks/src/imdb/run.rs
+++ b/benchmarks/src/imdb/run.rs
@@ -570,7 +570,7 @@ mod tests {
let plan = ctx.sql(&query).await?;
let plan = plan.create_physical_plan().await?;
let bytes = physical_plan_to_bytes(plan.clone())?;
- let plan2 = physical_plan_from_bytes(&bytes, &ctx)?;
+ let plan2 = physical_plan_from_bytes(&bytes, &ctx.task_ctx())?;
let plan_formatted = format!("{}", displayable(plan.as_ref()).indent(false));
let plan2_formatted =
format!("{}", displayable(plan2.as_ref()).indent(false));
diff --git a/benchmarks/src/lib.rs b/benchmarks/src/lib.rs
index 5d982fad6..07cffa5ae 100644
--- a/benchmarks/src/lib.rs
+++ b/benchmarks/src/lib.rs
@@ -19,6 +19,7 @@
pub mod cancellation;
pub mod clickbench;
pub mod h2o;
+pub mod hj;
pub mod imdb;
pub mod nlj;
pub mod sort_tpch;
diff --git a/benchmarks/src/tpch/run.rs b/benchmarks/src/tpch/run.rs
index 30ecb4d33..b93bdf254 100644
--- a/benchmarks/src/tpch/run.rs
+++ b/benchmarks/src/tpch/run.rs
@@ -424,7 +424,7 @@ mod tests {
let plan = ctx.sql(&query).await?;
let plan = plan.create_physical_plan().await?;
let bytes = physical_plan_to_bytes(plan.clone())?;
- let plan2 = physical_plan_from_bytes(&bytes, &ctx)?;
+ let plan2 = physical_plan_from_bytes(&bytes, &ctx.task_ctx())?;
let plan_formatted = format!("{}", displayable(plan.as_ref()).indent(false));
let plan2_formatted =
format!("{}", displayable(plan2.as_ref()).indent(false));
diff --git a/rust-toolchain.toml b/ci/scripts/license_header.sh
old mode 100644
new mode 100755
similarity index 82%
copy from rust-toolchain.toml
copy to ci/scripts/license_header.sh
index 55d572362..5345728f9
--- a/rust-toolchain.toml
+++ b/ci/scripts/license_header.sh
@@ -1,3 +1,5 @@
+#!/usr/bin/env bash
+#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,9 +17,6 @@
# specific language governing permissions and limitations
# under the License.
-# This file specifies the default version of Rust used
-# to compile this workspace and run CI jobs.
-
-[toolchain]
-channel = "1.89.0"
-components = ["rustfmt", "clippy"]
+# Check Apache license header
+set -ex
+hawkeye check --config licenserc.toml
diff --git a/datafusion-cli/CONTRIBUTING.md b/datafusion-cli/CONTRIBUTING.md
index 7c19c8e8b..8be656ec4 100644
--- a/datafusion-cli/CONTRIBUTING.md
+++ b/datafusion-cli/CONTRIBUTING.md
@@ -21,10 +21,16 @@
## Running Tests
-Tests can be run using `cargo`
+First check out test files with
```shell
-cargo test
+git submodule update --init
+```
+
+Then run all the tests with
+
+```shell
+cargo test --all-targets
```
## Running Storage Integration Tests
diff --git a/datafusion-cli/README.md b/datafusion-cli/README.md
index ca796b525..b34aa7703 100644
--- a/datafusion-cli/README.md
+++ b/datafusion-cli/README.md
@@ -19,12 +19,15 @@
<!-- Note this file is included in the crates.io page as well https://crates.io/crates/datafusion-cli -->
-# DataFusion Command-line Interface
+# Apache DataFusion Command-line Interface
-[DataFusion](https://datafusion.apache.org/) is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Apache DataFusion] is an extensible query execution framework, written in Rust, that uses [Apache Arrow] as its in-memory format.
DataFusion CLI (`datafusion-cli`) is a small command line utility that runs SQL queries using the DataFusion engine.
+[apache arrow]: https://arrow.apache.org/
+[apache datafusion]: https://datafusion.apache.org/
+
# Frequently Asked Questions
## Where can I find more information?
diff --git a/datafusion-cli/src/main.rs b/datafusion-cli/src/main.rs
index ab1726f22..a6b818c10 100644
--- a/datafusion-cli/src/main.rs
+++ b/datafusion-cli/src/main.rs
@@ -571,15 +571,15 @@ mod tests {
let df = ctx.sql(sql).await?;
let rbs = df.collect().await?;
- assert_snapshot!(batches_to_string(&rbs),@r#"
+ assert_snapshot!(batches_to_string(&rbs),@r"
+-----------------------------------+-----------------+---------------------+------+------------------+
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 10181 | 2 | page_index=false |
- | alltypes_tiny_pages.parquet | 454233 | 881634 | 2 | page_index=true |
+ | alltypes_tiny_pages.parquet | 454233 | 881418 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 2939 | 2 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
- "#);
+ ");
// increase the number of hits
ctx.sql("select * from alltypes_plain")
@@ -602,15 +602,15 @@ mod tests {
let df = ctx.sql(sql).await?;
let rbs = df.collect().await?;
- assert_snapshot!(batches_to_string(&rbs),@r#"
+ assert_snapshot!(batches_to_string(&rbs),@r"
+-----------------------------------+-----------------+---------------------+------+------------------+
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 10181 | 5 | page_index=false |
- | alltypes_tiny_pages.parquet | 454233 | 881634 | 2 | page_index=true |
+ | alltypes_tiny_pages.parquet | 454233 | 881418 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 2939 | 3 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
- "#);
+ ");
Ok(())
}
diff --git a/datafusion-examples/examples/composed_extension_codec.rs b/datafusion-examples/examples/composed_extension_codec.rs
index d3548167f..57f2c3704 100644
--- a/datafusion-examples/examples/composed_extension_codec.rs
+++ b/datafusion-examples/examples/composed_extension_codec.rs
@@ -32,12 +32,11 @@
use std::any::Any;
use std::fmt::Debug;
-use std::ops::Deref;
use std::sync::Arc;
use datafusion::common::internal_err;
use datafusion::common::Result;
-use datafusion::logical_expr::registry::FunctionRegistry;
+use datafusion::execution::TaskContext;
use datafusion::physical_plan::{DisplayAs, ExecutionPlan};
use datafusion::prelude::SessionContext;
use datafusion_proto::physical_plan::{
@@ -71,9 +70,8 @@ async fn main() {
.expect("to proto");
// deserialize proto back to execution plan
- let runtime = ctx.runtime_env();
let result_exec_plan: Arc<dyn ExecutionPlan> = proto
- .try_into_physical_plan(&ctx, runtime.deref(), &composed_codec)
+ .try_into_physical_plan(&ctx.task_ctx(), &composed_codec)
.expect("from proto");
// assert that the original and deserialized execution plans are equal
@@ -124,7 +122,7 @@ impl ExecutionPlan for ParentExec {
fn execute(
&self,
_partition: usize,
- _context: Arc<datafusion::execution::TaskContext>,
+ _context: Arc<TaskContext>,
) -> Result<datafusion::physical_plan::SendableRecordBatchStream> {
unreachable!()
}
@@ -139,7 +137,7 @@ impl PhysicalExtensionCodec for ParentPhysicalExtensionCodec {
&self,
buf: &[u8],
inputs: &[Arc<dyn ExecutionPlan>],
- _registry: &dyn FunctionRegistry,
+ _ctx: &TaskContext,
) -> Result<Arc<dyn ExecutionPlan>> {
if buf == "ParentExec".as_bytes() {
Ok(Arc::new(ParentExec {
@@ -200,7 +198,7 @@ impl ExecutionPlan for ChildExec {
fn execute(
&self,
_partition: usize,
- _context: Arc<datafusion::execution::TaskContext>,
+ _context: Arc<TaskContext>,
) -> Result<datafusion::physical_plan::SendableRecordBatchStream> {
unreachable!()
}
@@ -215,7 +213,7 @@ impl PhysicalExtensionCodec for ChildPhysicalExtensionCodec {
&self,
buf: &[u8],
_inputs: &[Arc<dyn ExecutionPlan>],
- _registry: &dyn FunctionRegistry,
+ _ctx: &TaskContext,
) -> Result<Arc<dyn ExecutionPlan>> {
if buf == "ChildExec".as_bytes() {
Ok(Arc::new(ChildExec {}))
diff --git a/datafusion-examples/examples/custom_file_casts.rs b/datafusion-examples/examples/custom_file_casts.rs
index e30ea1fb7..65ca09682 100644
--- a/datafusion-examples/examples/custom_file_casts.rs
+++ b/datafusion-examples/examples/custom_file_casts.rs
@@ -183,14 +183,18 @@ impl PhysicalExprAdapter for CustomCastsPhysicalExprAdapter {
// For example, [DataFusion Comet](https://github.com/apache/datafusion-comet) has a [custom cast kernel](https://github.com/apache/datafusion-comet/blob/b4ac876ab420ed403ac7fc8e1b29f42f1f442566/native/spark-expr/src/conversion_funcs/cast.rs#L133-L138).
expr.transform(|expr| {
if let Some(cast) = expr.as_any().downcast_ref::<CastExpr>() {
- let input_data_type = cast.expr().data_type(&self.physical_file_schema)?;
+ let input_data_type =
+ cast.expr().data_type(&self.physical_file_schema)?;
let output_data_type = cast.data_type(&self.physical_file_schema)?;
if !cast.is_bigger_cast(&input_data_type) {
- return not_impl_err!("Unsupported CAST from {input_data_type:?} to {output_data_type:?}")
+ return not_impl_err!(
+ "Unsupported CAST from {input_data_type} to {output_data_type}"
+ );
}
}
Ok(Transformed::no(expr))
- }).data()
+ })
+ .data()
}
fn with_partition_values(
diff --git a/datafusion-examples/examples/date_time_functions.rs b/datafusion-examples/examples/date_time_functions.rs
index dbe997043..2628319ae 100644
--- a/datafusion-examples/examples/date_time_functions.rs
+++ b/datafusion-examples/examples/date_time_functions.rs
@@ -492,14 +492,14 @@ async fn query_to_char() -> Result<()> {
assert_batches_eq!(
&[
- "+------------------------------+",
- "| to_char(t.values,t.patterns) |",
- "+------------------------------+",
- "| 2020-09-01 |",
- "| 2020:09:02 |",
- "| 20200903 |",
- "| 04-09-2020 |",
- "+------------------------------+",
+ "+----------------------------------+",
+ "| date_format(t.values,t.patterns) |",
+ "+----------------------------------+",
+ "| 2020-09-01 |",
+ "| 2020:09:02 |",
+ "| 20200903 |",
+ "| 04-09-2020 |",
+ "+----------------------------------+",
],
&result
);
diff --git a/datafusion-examples/examples/expr_api.rs b/datafusion-examples/examples/expr_api.rs
index ac17bd417..56f960870 100644
--- a/datafusion-examples/examples/expr_api.rs
+++ b/datafusion-examples/examples/expr_api.rs
@@ -519,7 +519,7 @@ fn type_coercion_demo() -> Result<()> {
)?;
let i8_array = Int8Array::from_iter_values(vec![0, 1, 2]);
let batch = RecordBatch::try_new(
- Arc::new(df_schema.as_arrow().to_owned()),
+ Arc::clone(df_schema.inner()),
vec![Arc::new(i8_array) as _],
)?;
diff --git a/datafusion-examples/examples/flight/flight_server.rs b/datafusion-examples/examples/flight/flight_server.rs
index cc5f43746..58bfb7a34 100644
--- a/datafusion-examples/examples/flight/flight_server.rs
+++ b/datafusion-examples/examples/flight/flight_server.rs
@@ -98,7 +98,7 @@ impl FlightService for FlightServiceImpl {
let df = ctx.sql(sql).await.map_err(to_tonic_err)?;
// execute the query
- let schema = df.schema().clone().into();
+ let schema = Arc::clone(df.schema().inner());
let results = df.collect().await.map_err(to_tonic_err)?;
if results.is_empty() {
return Err(Status::internal("There were no results from ticket"));
diff --git a/datafusion-examples/examples/flight/flight_sql_server.rs b/datafusion-examples/examples/flight/flight_sql_server.rs
index 5a573ed52..c35debec7 100644
--- a/datafusion-examples/examples/flight/flight_sql_server.rs
+++ b/datafusion-examples/examples/flight/flight_sql_server.rs
@@ -395,10 +395,8 @@ impl FlightSqlService for FlightSqlServiceImpl {
let plan_uuid = Uuid::new_v4().hyphenated().to_string();
self.statements.insert(plan_uuid.clone(), plan.clone());
- let plan_schema = plan.schema();
-
- let arrow_schema = (&**plan_schema).into();
- let message = SchemaAsIpc::new(&arrow_schema, &IpcWriteOptions::default())
+ let arrow_schema = plan.schema().as_arrow();
+ let message = SchemaAsIpc::new(arrow_schema, &IpcWriteOptions::default())
.try_into()
.map_err(|e| status!("Unable to serialize schema", e))?;
let IpcMessage(schema_bytes) = message;
diff --git a/datafusion/catalog-listing/Cargo.toml b/datafusion/catalog-listing/Cargo.toml
index b88461e7e..38b843dec 100644
--- a/datafusion/catalog-listing/Cargo.toml
+++ b/datafusion/catalog-listing/Cargo.toml
@@ -18,11 +18,11 @@
[package]
name = "datafusion-catalog-listing"
description = "datafusion-catalog-listing"
+readme = "README.md"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
-readme.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
diff --git a/datafusion/catalog-listing/README.md b/datafusion/catalog-listing/README.md
index c8d1cf13b..81a7c7b1d 100644
--- a/datafusion/catalog-listing/README.md
+++ b/datafusion/catalog-listing/README.md
@@ -17,9 +17,9 @@
under the License.
-->
-# DataFusion catalog-listing
+# Apache DataFusion Catalog Listing
-[DataFusion][df] is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Apache DataFusion] is an extensible query execution framework, written in Rust, that uses [Apache Arrow] as its in-memory format.
This crate is a submodule of DataFusion with [ListingTable], an implementation
of [TableProvider] based on files in a directory (either locally or on remote
@@ -29,8 +29,8 @@ Most projects should use the [`datafusion`] crate directly, which re-exports
this module. If you are already using the [`datafusion`] crate, there is no
reason to use this crate directly in your project as well.
-[df]: https://crates.io/crates/datafusion
-[df]: https://crates.io/crates/datafusion
+[apache arrow]: https://arrow.apache.org/
+[apache datafusion]: https://datafusion.apache.org/
[listingtable]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html
[tableprovider]: https://docs.rs/datafusion/latest/datafusion/datasource/trait.TableProvider.html
[`datafusion`]: https://crates.io/crates/datafusion
diff --git a/datafusion/catalog/Cargo.toml b/datafusion/catalog/Cargo.toml
index 5edb1c4a6..a1db45654 100644
--- a/datafusion/catalog/Cargo.toml
+++ b/datafusion/catalog/Cargo.toml
@@ -18,11 +18,11 @@
[package]
name = "datafusion-catalog"
description = "datafusion-catalog"
+readme = "README.md"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
-readme.workspace = true
repository.workspace = true
rust-version.workspace = true
version.workspace = true
diff --git a/datafusion/catalog/README.md b/datafusion/catalog/README.md
index d4870e28f..48c61b43c 100644
--- a/datafusion/catalog/README.md
+++ b/datafusion/catalog/README.md
@@ -17,9 +17,9 @@
under the License.
-->
-# DataFusion Catalog
+# Apache DataFusion Catalog
-[DataFusion][df] is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Apache DataFusion] is an extensible query execution framework, written in Rust, that uses [Apache Arrow] as its in-memory format.
This crate is a submodule of DataFusion that provides catalog management functionality, including catalogs, schemas, and tables.
@@ -27,5 +27,6 @@ Most projects should use the [`datafusion`] crate directly, which re-exports
this module. If you are already using the [`datafusion`] crate, there is no
reason to use this crate directly in your project as well.
-[df]: https://crates.io/crates/datafusion
+[apache arrow]: https://arrow.apache.org/
+[apache datafusion]: https://datafusion.apache.org/
[`datafusion`]: https://crates.io/crates/datafusion
diff --git a/datafusion/catalog/src/information_schema.rs b/datafusion/catalog/src/information_schema.rs
index 17dbc31d9..d733551f4 100644
--- a/datafusion/catalog/src/information_schema.rs
+++ b/datafusion/catalog/src/information_schema.rs
@@ -480,7 +480,7 @@ fn get_udwf_args_and_return_types(
#[inline]
fn remove_native_type_prefix(native_type: NativeType) -> String {
- format!("{native_type:?}")
+ format!("{native_type}")
}
#[async_trait]
@@ -827,8 +827,7 @@ impl InformationSchemaColumnsBuilder {
self.is_nullables.append_value(nullable_str);
// "System supplied type" --> Use debug format of the datatype
- self.data_types
- .append_value(format!("{:?}", field.data_type()));
+ self.data_types.append_value(field.data_type().to_string());
// "If data_type identifies a character or bit string type, the
// declared maximum length; null for all other data types or
diff --git a/datafusion/catalog/src/listing_schema.rs b/datafusion/catalog/src/listing_schema.rs
index 2e4eac964..7e19c1eca 100644
--- a/datafusion/catalog/src/listing_schema.rs
+++ b/datafusion/catalog/src/listing_schema.rs
@@ -136,6 +136,7 @@ impl ListingSchemaProvider {
file_type: self.format.clone(),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
diff --git a/datafusion/catalog/src/stream.rs b/datafusion/catalog/src/stream.rs
index 0fab9beba..2d66ff462 100644
--- a/datafusion/catalog/src/stream.rs
+++ b/datafusion/catalog/src/stream.rs
@@ -53,7 +53,7 @@ impl TableProviderFactory for StreamTableFactory {
state: &dyn Session,
cmd: &CreateExternalTable,
) -> Result<Arc<dyn TableProvider>> {
- let schema: SchemaRef = Arc::new(cmd.schema.as_ref().into());
+ let schema: SchemaRef = Arc::clone(cmd.schema.inner());
let location = cmd.location.clone();
let encoding = cmd.file_type.parse()?;
let header = if let Ok(opt) = cmd
diff --git a/datafusion/catalog/src/view.rs b/datafusion/catalog/src/view.rs
index 3bb721439..89c6a4a22 100644
--- a/datafusion/catalog/src/view.rs
+++ b/datafusion/catalog/src/view.rs
@@ -51,7 +51,7 @@ impl ViewTable {
/// Notes: the `LogicalPlan` is not validated or type coerced. If this is
/// needed it should be done after calling this function.
pub fn new(logical_plan: LogicalPlan, definition: Option<String>) -> Self {
- let table_schema = logical_plan.schema().as_ref().to_owned().into();
+ let table_schema = Arc::clone(logical_plan.schema().inner());
Self {
logical_plan,
table_schema,
diff --git a/datafusion/common-runtime/README.md b/datafusion/common-runtime/README.md
index bd0d4954b..ff44e6c3e 100644
--- a/datafusion/common-runtime/README.md
+++ b/datafusion/common-runtime/README.md
@@ -17,9 +17,9 @@
under the License.
-->
-# DataFusion Common Runtime
+# Apache DataFusion Common Runtime
-[DataFusion][df] is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Apache DataFusion] is an extensible query execution framework, written in Rust, that uses [Apache Arrow] as its in-memory format.
This crate is a submodule of DataFusion that provides common utilities.
@@ -27,5 +27,6 @@ Most projects should use the [`datafusion`] crate directly, which re-exports
this module. If you are already using the [`datafusion`] crate, there is no
reason to use this crate directly in your project as well.
-[df]: https://crates.io/crates/datafusion
+[apache arrow]: https://arrow.apache.org/
+[apache datafusion]: https://datafusion.apache.org/
[`datafusion`]: https://crates.io/crates/datafusion
diff --git a/datafusion/common/Cargo.toml b/datafusion/common/Cargo.toml
index d0e78e071..f5e51cb23 100644
--- a/datafusion/common/Cargo.toml
+++ b/datafusion/common/Cargo.toml
@@ -66,7 +66,7 @@ half = { workspace = true }
hashbrown = { workspace = true }
hex = { workspace = true, optional = true }
indexmap = { workspace = true }
-libc = "0.2.175"
+libc = "0.2.176"
log = { workspace = true }
object_store = { workspace = true, optional = true }
parquet = { workspace = true, optional = true, default-features = true }
diff --git a/datafusion/common/README.md b/datafusion/common/README.md
index e4d6b7726..4948c8c58 100644
--- a/datafusion/common/README.md
+++ b/datafusion/common/README.md
@@ -17,9 +17,9 @@
under the License.
-->
-# DataFusion Common
+# Apache DataFusion Common
-[DataFusion][df] is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Apache DataFusion] is an extensible query execution framework, written in Rust, that uses [Apache Arrow] as its in-memory format.
This crate is a submodule of DataFusion that provides common data types and utilities.
@@ -27,5 +27,6 @@ Most projects should use the [`datafusion`] crate directly, which re-exports
this module. If you are already using the [`datafusion`] crate, there is no
reason to use this crate directly in your project as well.
-[df]: https://crates.io/crates/datafusion
+[apache arrow]: https://arrow.apache.org/
+[apache datafusion]: https://datafusion.apache.org/
[`datafusion`]: https://crates.io/crates/datafusion
diff --git a/datafusion/common/src/cast.rs b/datafusion/common/src/cast.rs
index 68b753a66..e6eda3c58 100644
--- a/datafusion/common/src/cast.rs
+++ b/datafusion/common/src/cast.rs
@@ -22,9 +22,10 @@
use crate::{downcast_value, Result};
use arrow::array::{
- BinaryViewArray, DurationMicrosecondArray, DurationMillisecondArray,
- DurationNanosecondArray, DurationSecondArray, Float16Array, Int16Array, Int8Array,
- LargeBinaryArray, LargeStringArray, StringViewArray, UInt16Array,
+ BinaryViewArray, Decimal32Array, Decimal64Array, DurationMicrosecondArray,
+ DurationMillisecondArray, DurationNanosecondArray, DurationSecondArray, Float16Array,
+ Int16Array, Int8Array, LargeBinaryArray, LargeStringArray, StringViewArray,
+ UInt16Array,
};
use arrow::{
array::{
@@ -97,6 +98,16 @@ pub fn as_uint64_array(array: &dyn Array) -> Result<&UInt64Array> {
Ok(downcast_value!(array, UInt64Array))
}
+// Downcast Array to Decimal32Array
+pub fn as_decimal32_array(array: &dyn Array) -> Result<&Decimal32Array> {
+ Ok(downcast_value!(array, Decimal32Array))
+}
+
+// Downcast Array to Decimal64Array
+pub fn as_decimal64_array(array: &dyn Array) -> Result<&Decimal64Array> {
+ Ok(downcast_value!(array, Decimal64Array))
+}
+
// Downcast Array to Decimal128Array
pub fn as_decimal128_array(array: &dyn Array) -> Result<&Decimal128Array> {
Ok(downcast_value!(array, Decimal128Array))
@@ -302,7 +313,7 @@ pub fn as_fixed_size_list_array(array: &dyn Array) -> Result<&FixedSizeListArray
Ok(downcast_value!(array, FixedSizeListArray))
}
-// Downcast Array to FixedSizeListArray
+// Downcast Array to FixedSizeBinaryArray
pub fn as_fixed_size_binary_array(array: &dyn Array) -> Result<&FixedSizeBinaryArray> {
Ok(downcast_value!(array, FixedSizeBinaryArray))
}
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 68968e729..adfb1a6ef 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -566,6 +566,14 @@ config_namespace! {
/// (reading) Use any available bloom filters when reading parquet files
pub bloom_filter_on_read: bool, default = true
+ /// (reading) The maximum predicate cache size, in bytes. When
+ /// `pushdown_filters` is enabled, sets the maximum memory used to cache
+ /// the results of predicate evaluation between filter evaluation and
+ /// output generation. Decreasing this value will reduce memory usage,
+ /// but may increase IO and CPU usage. None means use the default
+ /// parquet reader setting. 0 means no caching.
+ pub max_predicate_cache_size: Option<usize>, default = None
+
// The following options affect writing to parquet files
// and map to parquet::file::properties::WriterProperties
diff --git a/datafusion/common/src/dfschema.rs b/datafusion/common/src/dfschema.rs
index 48d255503..b195b1d4a 100644
--- a/datafusion/common/src/dfschema.rs
+++ b/datafusion/common/src/dfschema.rs
@@ -101,7 +101,7 @@ pub type DFSchemaRef = Arc<DFSchema>;
/// let df_schema = DFSchema::from_unqualified_fields(vec![
/// Field::new("c1", arrow::datatypes::DataType::Int32, false),
/// ].into(),HashMap::new()).unwrap();
-/// let schema = Schema::from(df_schema);
+/// let schema: &Schema = df_schema.as_arrow();
/// assert_eq!(schema.fields().len(), 1);
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -594,7 +594,7 @@ impl DFSchema {
&self,
arrow_schema: &Schema,
) -> Result<()> {
- let self_arrow_schema: Schema = self.into();
+ let self_arrow_schema = self.as_arrow();
self_arrow_schema
.fields()
.iter()
@@ -669,8 +669,8 @@ impl DFSchema {
))
{
_plan_err!(
- "Schema mismatch: Expected field '{}' with type {:?}, \
- but got '{}' with type {:?}.",
+ "Schema mismatch: Expected field '{}' with type {}, \
+ but got '{}' with type {}.",
f1.name(),
f1.data_type(),
f2.name(),
@@ -747,7 +747,8 @@ impl DFSchema {
}
/// Returns true of two [`DataType`]s are semantically equal (same
- /// name and type), ignoring both metadata and nullability, and decimal precision/scale.
+ /// name and type), ignoring both metadata and nullability, decimal precision/scale,
+ /// and timezone time units/timezones.
///
/// request to upstream: <https://github.com/apache/arrow-rs/issues/3199>
pub fn datatype_is_semantically_equal(dt1: &DataType, dt2: &DataType) -> bool {
@@ -798,6 +799,14 @@ impl DFSchema {
.zip(iter2)
.all(|((t1, f1), (t2, f2))| t1 == t2 && Self::field_is_semantically_equal(f1, f2))
}
+ (
+ DataType::Decimal32(_l_precision, _l_scale),
+ DataType::Decimal32(_r_precision, _r_scale),
+ ) => true,
+ (
+ DataType::Decimal64(_l_precision, _l_scale),
+ DataType::Decimal64(_r_precision, _r_scale),
+ ) => true,
(
DataType::Decimal128(_l_precision, _l_scale),
DataType::Decimal128(_r_precision, _r_scale),
@@ -806,6 +815,10 @@ impl DFSchema {
DataType::Decimal256(_l_precision, _l_scale),
DataType::Decimal256(_r_precision, _r_scale),
) => true,
+ (
+ DataType::Timestamp(_l_time_unit, _l_timezone),
+ DataType::Timestamp(_r_time_unit, _r_timezone),
+ ) => true,
_ => dt1 == dt2,
}
}
@@ -1056,6 +1069,12 @@ fn format_simple_data_type(data_type: &DataType) -> String {
DataType::Dictionary(_, value_type) => {
format_simple_data_type(value_type.as_ref())
}
+ DataType::Decimal32(precision, scale) => {
+ format!("decimal32({precision}, {scale})")
+ }
+ DataType::Decimal64(precision, scale) => {
+ format!("decimal64({precision}, {scale})")
+ }
DataType::Decimal128(precision, scale) => {
format!("decimal128({precision}, {scale})")
}
@@ -1063,23 +1082,7 @@ fn format_simple_data_type(data_type: &DataType) -> String {
format!("decimal256({precision}, {scale})")
}
DataType::Null => "null".to_string(),
- _ => format!("{data_type:?}").to_lowercase(),
- }
-}
-
-impl From<DFSchema> for Schema {
- /// Convert DFSchema into a Schema
- fn from(df_schema: DFSchema) -> Self {
- let fields: Fields = df_schema.inner.fields.clone();
- Schema::new_with_metadata(fields, df_schema.inner.metadata.clone())
- }
-}
-
-impl From<&DFSchema> for Schema {
- /// Convert DFSchema reference into a Schema
- fn from(df_schema: &DFSchema) -> Self {
- let fields: Fields = df_schema.inner.fields.clone();
- Schema::new_with_metadata(fields, df_schema.inner.metadata.clone())
+ _ => format!("{data_type}").to_lowercase(),
}
}
@@ -1115,17 +1118,15 @@ impl TryFrom<SchemaRef> for DFSchema {
field_qualifiers: vec![None; field_count],
functional_dependencies: FunctionalDependencies::empty(),
};
- dfschema.check_names()?;
+ // Without checking names, because schema here may have duplicate field names.
+ // For example, Partial AggregateMode will generate duplicate field names from
+ // state_fields.
+ // See <https://github.com/apache/datafusion/issues/17715>
+ // dfschema.check_names()?;
Ok(dfschema)
}
}
-impl From<DFSchema> for SchemaRef {
- fn from(df_schema: DFSchema) -> Self {
- SchemaRef::new(df_schema.into())
- }
-}
-
// Hashing refers to a subset of fields considered in PartialEq.
impl Hash for DFSchema {
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
@@ -1308,8 +1309,8 @@ impl SchemaExt for Schema {
.try_for_each(|(f1, f2)| {
if f1.name() != f2.name() || (!DFSchema::datatype_is_logically_equal(f1.data_type(), f2.data_type()) && !can_cast_types(f2.data_type(), f1.data_type())) {
_plan_err!(
- "Inserting query schema mismatch: Expected table field '{}' with type {:?}, \
- but got '{}' with type {:?}.",
+ "Inserting query schema mismatch: Expected table field '{}' with type {}, \
+ but got '{}' with type {}.",
f1.name(),
f1.data_type(),
f2.name(),
@@ -1415,7 +1416,7 @@ mod tests {
#[test]
fn from_qualified_schema_into_arrow_schema() -> Result<()> {
let schema = DFSchema::try_from_qualified_schema("t1", &test_schema_1())?;
- let arrow_schema: Schema = schema.into();
+ let arrow_schema = schema.as_arrow();
let expected = "Field { name: \"c0\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, \
Field { name: \"c1\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }";
assert_eq!(expected, arrow_schema.to_string());
@@ -1794,6 +1795,36 @@ mod tests {
&DataType::Int16
));
+ // Succeeds if decimal precision and scale are different
+ assert!(DFSchema::datatype_is_semantically_equal(
+ &DataType::Decimal32(1, 2),
+ &DataType::Decimal32(2, 1),
+ ));
+
+ assert!(DFSchema::datatype_is_semantically_equal(
+ &DataType::Decimal64(1, 2),
+ &DataType::Decimal64(2, 1),
+ ));
+
+ assert!(DFSchema::datatype_is_semantically_equal(
+ &DataType::Decimal128(1, 2),
+ &DataType::Decimal128(2, 1),
+ ));
+
+ assert!(DFSchema::datatype_is_semantically_equal(
+ &DataType::Decimal256(1, 2),
+ &DataType::Decimal256(2, 1),
+ ));
+
+ // Any two timestamp types should match
+ assert!(DFSchema::datatype_is_semantically_equal(
+ &DataType::Timestamp(
+ arrow::datatypes::TimeUnit::Microsecond,
+ Some("UTC".into())
+ ),
+ &DataType::Timestamp(arrow::datatypes::TimeUnit::Millisecond, None),
+ ));
+
// Test lists
// Succeeds if both have the same element type, disregards names and nullability
@@ -2377,6 +2408,8 @@ mod tests {
),
false,
),
+ Field::new("decimal32", DataType::Decimal32(9, 4), true),
+ Field::new("decimal64", DataType::Decimal64(9, 4), true),
Field::new("decimal128", DataType::Decimal128(18, 4), true),
Field::new("decimal256", DataType::Decimal256(38, 10), false),
Field::new("date32", DataType::Date32, true),
@@ -2408,6 +2441,8 @@ mod tests {
|-- fixed_size_binary: fixed_size_binary (nullable = true)
|-- fixed_size_list: fixed size list (nullable = false)
| |-- item: int32 (nullable = true)
+ |-- decimal32: decimal32(9, 4) (nullable = true)
+ |-- decimal64: decimal64(9, 4) (nullable = true)
|-- decimal128: decimal128(18, 4) (nullable = true)
|-- decimal256: decimal256(38, 10) (nullable = false)
|-- date32: date32 (nullable = true)
diff --git a/datafusion/common/src/file_options/parquet_writer.rs b/datafusion/common/src/file_options/parquet_writer.rs
index a48700a9e..3977f2b48 100644
--- a/datafusion/common/src/file_options/parquet_writer.rs
+++ b/datafusion/common/src/file_options/parquet_writer.rs
@@ -208,6 +208,7 @@ impl ParquetOptions {
binary_as_string: _, // not used for writer props
coerce_int96: _, // not used for writer props
skip_arrow_metadata: _,
+ max_predicate_cache_size: _,
} = self;
let mut builder = WriterProperties::builder()
@@ -400,6 +401,10 @@ pub(crate) fn parse_statistics_string(str_setting: &str) -> Result<EnabledStatis
#[cfg(feature = "parquet")]
#[cfg(test)]
mod tests {
+ use super::*;
+ use crate::config::{ParquetColumnOptions, ParquetEncryptionOptions, ParquetOptions};
+ #[cfg(feature = "parquet_encryption")]
+ use crate::encryption::map_encryption_to_config_encryption;
use parquet::{
basic::Compression,
file::properties::{
@@ -409,11 +414,6 @@ mod tests {
};
use std::collections::HashMap;
- use super::*;
- use crate::config::{ParquetColumnOptions, ParquetEncryptionOptions, ParquetOptions};
- #[cfg(feature = "parquet_encryption")]
- use crate::encryption::map_encryption_to_config_encryption;
-
const COL_NAME: &str = "configured";
/// Take the column defaults provided in [`ParquetOptions`], and generate a non-default col config.
@@ -475,6 +475,7 @@ mod tests {
binary_as_string: defaults.binary_as_string,
skip_arrow_metadata: defaults.skip_arrow_metadata,
coerce_int96: None,
+ max_predicate_cache_size: defaults.max_predicate_cache_size,
}
}
@@ -581,6 +582,8 @@ mod tests {
maximum_buffered_record_batches_per_stream: global_options_defaults
.maximum_buffered_record_batches_per_stream,
bloom_filter_on_read: global_options_defaults.bloom_filter_on_read,
+ max_predicate_cache_size: global_options_defaults
+ .max_predicate_cache_size,
schema_force_view_types: global_options_defaults.schema_force_view_types,
binary_as_string: global_options_defaults.binary_as_string,
skip_arrow_metadata: global_options_defaults.skip_arrow_metadata,
diff --git a/datafusion/common/src/nested_struct.rs b/datafusion/common/src/nested_struct.rs
index 6e8a380df..38060e370 100644
--- a/datafusion/common/src/nested_struct.rs
+++ b/datafusion/common/src/nested_struct.rs
@@ -87,7 +87,7 @@ fn cast_struct_column(
} else {
// Return error if source is not a struct type
_plan_err!(
- "Cannot cast column of type {:?} to struct type. Source must be a struct to cast to struct.",
+ "Cannot cast column of type {} to struct type. Source must be a struct to cast to struct.",
source_col.data_type()
)
}
@@ -128,7 +128,7 @@ fn cast_struct_column(
/// The struct casting logic requires that the source column must already be a struct type.
/// This makes the function useful for:
/// - Schema evolution scenarios where struct layouts change over time
-/// - Data migration between different struct schemas
+/// - Data migration between different struct schemas
/// - Type-safe data processing pipelines that maintain struct type integrity
///
/// # Arguments
@@ -165,7 +165,7 @@ pub fn cast_column(
/// Validates compatibility between source and target struct fields for casting operations.
///
/// This function implements comprehensive struct compatibility checking by examining:
-/// - Field name matching between source and target structs
+/// - Field name matching between source and target structs
/// - Type castability for each matching field (including recursive struct validation)
/// - Proper handling of missing fields (target fields not in source are allowed - filled with nulls)
/// - Proper handling of extra fields (source fields not in target are allowed - ignored)
@@ -173,7 +173,7 @@ pub fn cast_column(
/// # Compatibility Rules
/// - **Field Matching**: Fields are matched by name (case-sensitive)
/// - **Missing Target Fields**: Allowed - will be filled with null values during casting
-/// - **Extra Source Fields**: Allowed - will be ignored during casting
+/// - **Extra Source Fields**: Allowed - will be ignored during casting
/// - **Type Compatibility**: Each matching field must be castable using Arrow's type system
/// - **Nested Structs**: Recursively validates nested struct compatibility
///
@@ -188,7 +188,7 @@ pub fn cast_column(
/// # Examples
/// ```text
/// // Compatible: source has extra field, target has missing field
-/// // Source: {a: i32, b: string, c: f64}
+/// // Source: {a: i32, b: string, c: f64}
/// // Target: {a: i64, d: bool}
/// // Result: Ok(()) - 'a' can cast i32->i64, 'b','c' ignored, 'd' filled with nulls
///
@@ -230,7 +230,7 @@ pub fn validate_struct_compatibility(
target_field.data_type(),
) {
return _plan_err!(
- "Cannot cast struct field '{}' from type {:?} to type {:?}",
+ "Cannot cast struct field '{}' from type {} to type {}",
target_field.name(),
source_field.data_type(),
target_field.data_type()
diff --git a/datafusion/common/src/param_value.rs b/datafusion/common/src/param_value.rs
index d2802c096..7582cff56 100644
--- a/datafusion/common/src/param_value.rs
+++ b/datafusion/common/src/param_value.rs
@@ -48,7 +48,7 @@ impl ParamValues {
for (i, (param_type, value)) in iter.enumerate() {
if *param_type != value.data_type() {
return _plan_err!(
- "Expected parameter of type {:?}, got {:?} at index {}",
+ "Expected parameter of type {}, got {:?} at index {}",
param_type,
value.data_type(),
i
diff --git a/datafusion/common/src/scalar/mod.rs b/datafusion/common/src/scalar/mod.rs
index c5e764272..8c079056e 100644
--- a/datafusion/common/src/scalar/mod.rs
+++ b/datafusion/common/src/scalar/mod.rs
@@ -35,13 +35,14 @@ use std::sync::Arc;
use crate::cast::{
as_binary_array, as_binary_view_array, as_boolean_array, as_date32_array,
- as_date64_array, as_decimal128_array, as_decimal256_array, as_dictionary_array,
- as_duration_microsecond_array, as_duration_millisecond_array,
- as_duration_nanosecond_array, as_duration_second_array, as_fixed_size_binary_array,
- as_fixed_size_list_array, as_float16_array, as_float32_array, as_float64_array,
- as_int16_array, as_int32_array, as_int64_array, as_int8_array, as_interval_dt_array,
- as_interval_mdn_array, as_interval_ym_array, as_large_binary_array,
- as_large_list_array, as_large_string_array, as_string_array, as_string_view_array,
+ as_date64_array, as_decimal128_array, as_decimal256_array, as_decimal32_array,
+ as_decimal64_array, as_dictionary_array, as_duration_microsecond_array,
+ as_duration_millisecond_array, as_duration_nanosecond_array,
+ as_duration_second_array, as_fixed_size_binary_array, as_fixed_size_list_array,
+ as_float16_array, as_float32_array, as_float64_array, as_int16_array, as_int32_array,
+ as_int64_array, as_int8_array, as_interval_dt_array, as_interval_mdn_array,
+ as_interval_ym_array, as_large_binary_array, as_large_list_array,
+ as_large_string_array, as_string_array, as_string_view_array,
as_time32_millisecond_array, as_time32_second_array, as_time64_microsecond_array,
as_time64_nanosecond_array, as_timestamp_microsecond_array,
as_timestamp_millisecond_array, as_timestamp_nanosecond_array,
@@ -56,13 +57,14 @@ use crate::{_internal_datafusion_err, arrow_datafusion_err};
use arrow::array::{
new_empty_array, new_null_array, Array, ArrayData, ArrayRef, ArrowNativeTypeOp,
ArrowPrimitiveType, AsArray, BinaryArray, BinaryViewArray, BooleanArray, Date32Array,
- Date64Array, Decimal128Array, Decimal256Array, DictionaryArray,
- DurationMicrosecondArray, DurationMillisecondArray, DurationNanosecondArray,
- DurationSecondArray, FixedSizeBinaryArray, FixedSizeListArray, Float16Array,
- Float32Array, Float64Array, GenericListArray, Int16Array, Int32Array, Int64Array,
- Int8Array, IntervalDayTimeArray, IntervalMonthDayNanoArray, IntervalYearMonthArray,
- LargeBinaryArray, LargeListArray, LargeStringArray, ListArray, MapArray,
- MutableArrayData, PrimitiveArray, Scalar, StringArray, StringViewArray, StructArray,
+ Date64Array, Decimal128Array, Decimal256Array, Decimal32Array, Decimal64Array,
+ DictionaryArray, DurationMicrosecondArray, DurationMillisecondArray,
+ DurationNanosecondArray, DurationSecondArray, FixedSizeBinaryArray,
+ FixedSizeListArray, Float16Array, Float32Array, Float64Array, GenericListArray,
+ Int16Array, Int32Array, Int64Array, Int8Array, IntervalDayTimeArray,
+ IntervalMonthDayNanoArray, IntervalYearMonthArray, LargeBinaryArray, LargeListArray,
+ LargeStringArray, ListArray, MapArray, MutableArrayData, OffsetSizeTrait,
+ PrimitiveArray, Scalar, StringArray, StringViewArray, StructArray,
Time32MillisecondArray, Time32SecondArray, Time64MicrosecondArray,
Time64NanosecondArray, TimestampMicrosecondArray, TimestampMillisecondArray,
TimestampNanosecondArray, TimestampSecondArray, UInt16Array, UInt32Array,
@@ -75,12 +77,13 @@ use arrow::compute::kernels::numeric::{
};
use arrow::datatypes::{
i256, validate_decimal_precision_and_scale, ArrowDictionaryKeyType, ArrowNativeType,
- ArrowTimestampType, DataType, Date32Type, Decimal128Type, Decimal256Type, Field,
- Float32Type, Int16Type, Int32Type, Int64Type, Int8Type, IntervalDayTime,
- IntervalDayTimeType, IntervalMonthDayNano, IntervalMonthDayNanoType, IntervalUnit,
- IntervalYearMonthType, TimeUnit, TimestampMicrosecondType, TimestampMillisecondType,
- TimestampNanosecondType, TimestampSecondType, UInt16Type, UInt32Type, UInt64Type,
- UInt8Type, UnionFields, UnionMode, DECIMAL128_MAX_PRECISION,
+ ArrowTimestampType, DataType, Date32Type, Decimal128Type, Decimal256Type,
+ Decimal32Type, Decimal64Type, Field, Float32Type, Int16Type, Int32Type, Int64Type,
+ Int8Type, IntervalDayTime, IntervalDayTimeType, IntervalMonthDayNano,
+ IntervalMonthDayNanoType, IntervalUnit, IntervalYearMonthType, TimeUnit,
+ TimestampMicrosecondType, TimestampMillisecondType, TimestampNanosecondType,
+ TimestampSecondType, UInt16Type, UInt32Type, UInt64Type, UInt8Type, UnionFields,
+ UnionMode, DECIMAL128_MAX_PRECISION,
};
use arrow::util::display::{array_value_to_string, ArrayFormatter, FormatOptions};
use cache::{get_or_create_cached_key_array, get_or_create_cached_null_array};
@@ -231,6 +234,10 @@ pub enum ScalarValue {
Float32(Option<f32>),
/// 64bit float
Float64(Option<f64>),
+ /// 32bit decimal, using the i32 to represent the decimal, precision scale
+ Decimal32(Option<i32>, u8, i8),
+ /// 64bit decimal, using the i64 to represent the decimal, precision scale
+ Decimal64(Option<i64>, u8, i8),
/// 128bit decimal, using the i128 to represent the decimal, precision scale
Decimal128(Option<i128>, u8, i8),
/// 256bit decimal, using the i256 to represent the decimal, precision scale
@@ -340,6 +347,14 @@ impl PartialEq for ScalarValue {
// any newly added enum variant will require editing this list
// or else face a compile error
match (self, other) {
+ (Decimal32(v1, p1, s1), Decimal32(v2, p2, s2)) => {
+ v1.eq(v2) && p1.eq(p2) && s1.eq(s2)
+ }
+ (Decimal32(_, _, _), _) => false,
+ (Decimal64(v1, p1, s1), Decimal64(v2, p2, s2)) => {
+ v1.eq(v2) && p1.eq(p2) && s1.eq(s2)
+ }
+ (Decimal64(_, _, _), _) => false,
(Decimal128(v1, p1, s1), Decimal128(v2, p2, s2)) => {
v1.eq(v2) && p1.eq(p2) && s1.eq(s2)
}
@@ -459,6 +474,24 @@ impl PartialOrd for ScalarValue {
// any newly added enum variant will require editing this list
// or else face a compile error
match (self, other) {
+ (Decimal32(v1, p1, s1), Decimal32(v2, p2, s2)) => {
+ if p1.eq(p2) && s1.eq(s2) {
+ v1.partial_cmp(v2)
+ } else {
+ // Two decimal values can be compared if they have the same precision and scale.
+ None
+ }
+ }
+ (Decimal32(_, _, _), _) => None,
+ (Decimal64(v1, p1, s1), Decimal64(v2, p2, s2)) => {
+ if p1.eq(p2) && s1.eq(s2) {
+ v1.partial_cmp(v2)
+ } else {
+ // Two decimal values can be compared if they have the same precision and scale.
+ None
+ }
+ }
+ (Decimal64(_, _, _), _) => None,
(Decimal128(v1, p1, s1), Decimal128(v2, p2, s2)) => {
if p1.eq(p2) && s1.eq(s2) {
v1.partial_cmp(v2)
@@ -760,6 +793,16 @@ impl Hash for ScalarValue {
fn hash<H: Hasher>(&self, state: &mut H) {
use ScalarValue::*;
match self {
+ Decimal32(v, p, s) => {
+ v.hash(state);
+ p.hash(state);
+ s.hash(state)
+ }
+ Decimal64(v, p, s) => {
+ v.hash(state);
+ p.hash(state);
+ s.hash(state)
+ }
Decimal128(v, p, s) => {
v.hash(state);
p.hash(state);
@@ -834,8 +877,9 @@ impl Hash for ScalarValue {
}
fn hash_nested_array<H: Hasher>(arr: ArrayRef, state: &mut H) {
- let arrays = vec![arr.to_owned()];
- let hashes_buffer = &mut vec![0; arr.len()];
+ let len = arr.len();
+ let arrays = vec![arr];
+ let hashes_buffer = &mut vec![0; len];
let random_state = ahash::RandomState::with_seeds(0, 0, 0, 0);
let hashes = create_hashes(&arrays, &random_state, hashes_buffer).unwrap();
// Hash back to std::hash::Hasher
@@ -1045,6 +1089,12 @@ impl ScalarValue {
DataType::UInt16 => ScalarValue::UInt16(None),
DataType::UInt32 => ScalarValue::UInt32(None),
DataType::UInt64 => ScalarValue::UInt64(None),
+ DataType::Decimal32(precision, scale) => {
+ ScalarValue::Decimal32(None, *precision, *scale)
+ }
+ DataType::Decimal64(precision, scale) => {
+ ScalarValue::Decimal64(None, *precision, *scale)
+ }
DataType::Decimal128(precision, scale) => {
ScalarValue::Decimal128(None, *precision, *scale)
}
@@ -1137,7 +1187,7 @@ impl ScalarValue {
DataType::Null => ScalarValue::Null,
_ => {
return _not_impl_err!(
- "Can't create a null scalar from data_type \"{data_type:?}\""
+ "Can't create a null scalar from data_type \"{data_type}\""
);
}
})
@@ -1193,7 +1243,7 @@ impl ScalarValue {
match datatype {
DataType::Float32 => Ok(ScalarValue::from(std::f32::consts::PI)),
DataType::Float64 => Ok(ScalarValue::from(std::f64::consts::PI)),
- _ => _internal_err!("PI is not supported for data type: {:?}", datatype),
+ _ => _internal_err!("PI is not supported for data type: {}", datatype),
}
}
@@ -1203,7 +1253,7 @@ impl ScalarValue {
DataType::Float32 => Ok(ScalarValue::from(consts::PI_UPPER_F32)),
DataType::Float64 => Ok(ScalarValue::from(consts::PI_UPPER_F64)),
_ => {
- _internal_err!("PI_UPPER is not supported for data type: {:?}", datatype)
+ _internal_err!("PI_UPPER is not supported for data type: {}", datatype)
}
}
}
@@ -1214,7 +1264,7 @@ impl ScalarValue {
DataType::Float32 => Ok(ScalarValue::from(consts::NEGATIVE_PI_LOWER_F32)),
DataType::Float64 => Ok(ScalarValue::from(consts::NEGATIVE_PI_LOWER_F64)),
_ => {
- _internal_err!("-PI_LOWER is not supported for data type: {:?}", datatype)
+ _internal_err!("-PI_LOWER is not supported for data type: {}", datatype)
}
}
}
@@ -1225,10 +1275,7 @@ impl ScalarValue {
DataType::Float32 => Ok(ScalarValue::from(consts::FRAC_PI_2_UPPER_F32)),
DataType::Float64 => Ok(ScalarValue::from(consts::FRAC_PI_2_UPPER_F64)),
_ => {
- _internal_err!(
- "PI_UPPER/2 is not supported for data type: {:?}",
- datatype
- )
+ _internal_err!("PI_UPPER/2 is not supported for data type: {}", datatype)
}
}
}
@@ -1243,10 +1290,7 @@ impl ScalarValue {
Ok(ScalarValue::from(consts::NEGATIVE_FRAC_PI_2_LOWER_F64))
}
_ => {
- _internal_err!(
- "-PI/2_LOWER is not supported for data type: {:?}",
- datatype
- )
+ _internal_err!("-PI/2_LOWER is not supported for data type: {}", datatype)
}
}
}
@@ -1256,7 +1300,7 @@ impl ScalarValue {
match datatype {
DataType::Float32 => Ok(ScalarValue::from(-std::f32::consts::PI)),
DataType::Float64 => Ok(ScalarValue::from(-std::f64::consts::PI)),
- _ => _internal_err!("-PI is not supported for data type: {:?}", datatype),
+ _ => _internal_err!("-PI is not supported for data type: {}", datatype),
}
}
@@ -1265,7 +1309,7 @@ impl ScalarValue {
match datatype {
DataType::Float32 => Ok(ScalarValue::from(std::f32::consts::FRAC_PI_2)),
DataType::Float64 => Ok(ScalarValue::from(std::f64::consts::FRAC_PI_2)),
- _ => _internal_err!("PI/2 is not supported for data type: {:?}", datatype),
+ _ => _internal_err!("PI/2 is not supported for data type: {}", datatype),
}
}
@@ -1274,7 +1318,7 @@ impl ScalarValue {
match datatype {
DataType::Float32 => Ok(ScalarValue::from(-std::f32::consts::FRAC_PI_2)),
DataType::Float64 => Ok(ScalarValue::from(-std::f64::consts::FRAC_PI_2)),
- _ => _internal_err!("-PI/2 is not supported for data type: {:?}", datatype),
+ _ => _internal_err!("-PI/2 is not supported for data type: {}", datatype),
}
}
@@ -1284,7 +1328,7 @@ impl ScalarValue {
DataType::Float32 => Ok(ScalarValue::from(f32::INFINITY)),
DataType::Float64 => Ok(ScalarValue::from(f64::INFINITY)),
_ => {
- _internal_err!("Infinity is not supported for data type: {:?}", datatype)
+ _internal_err!("Infinity is not supported for data type: {}", datatype)
}
}
}
@@ -1296,7 +1340,7 @@ impl ScalarValue {
DataType::Float64 => Ok(ScalarValue::from(f64::NEG_INFINITY)),
_ => {
_internal_err!(
- "Negative Infinity is not supported for data type: {:?}",
+ "Negative Infinity is not supported for data type: {}",
datatype
)
}
@@ -1369,7 +1413,7 @@ impl ScalarValue {
DataType::Date64 => ScalarValue::Date64(Some(0)),
_ => {
return _not_impl_err!(
- "Can't create a zero scalar from data_type \"{datatype:?}\""
+ "Can't create a zero scalar from data_type \"{datatype}\""
);
}
})
@@ -1507,7 +1551,7 @@ impl ScalarValue {
// Unsupported types for now
_ => {
_not_impl_err!(
- "Default value for data_type \"{datatype:?}\" is not implemented yet"
+ "Default value for data_type \"{datatype}\" is not implemented yet"
)
}
}
@@ -1527,6 +1571,34 @@ impl ScalarValue {
DataType::Float16 => ScalarValue::Float16(Some(f16::from_f32(1.0))),
DataType::Float32 => ScalarValue::Float32(Some(1.0)),
DataType::Float64 => ScalarValue::Float64(Some(1.0)),
+ DataType::Decimal32(precision, scale) => {
+ validate_decimal_precision_and_scale::<Decimal32Type>(
+ *precision, *scale,
+ )?;
+ if *scale < 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match 10_i32.checked_pow(*scale as u32) {
+ Some(value) => {
+ ScalarValue::Decimal32(Some(value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
+ DataType::Decimal64(precision, scale) => {
+ validate_decimal_precision_and_scale::<Decimal64Type>(
+ *precision, *scale,
+ )?;
+ if *scale < 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match i64::from(10).checked_pow(*scale as u32) {
+ Some(value) => {
+ ScalarValue::Decimal64(Some(value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
DataType::Decimal128(precision, scale) => {
validate_decimal_precision_and_scale::<Decimal128Type>(
*precision, *scale,
@@ -1557,7 +1629,7 @@ impl ScalarValue {
}
_ => {
return _not_impl_err!(
- "Can't create an one scalar from data_type \"{datatype:?}\""
+ "Can't create an one scalar from data_type \"{datatype}\""
);
}
})
@@ -1573,6 +1645,34 @@ impl ScalarValue {
DataType::Float16 => ScalarValue::Float16(Some(f16::from_f32(-1.0))),
DataType::Float32 => ScalarValue::Float32(Some(-1.0)),
DataType::Float64 => ScalarValue::Float64(Some(-1.0)),
+ DataType::Decimal32(precision, scale) => {
+ validate_decimal_precision_and_scale::<Decimal32Type>(
+ *precision, *scale,
+ )?;
+ if *scale < 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match 10_i32.checked_pow(*scale as u32) {
+ Some(value) => {
+ ScalarValue::Decimal32(Some(-value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
+ DataType::Decimal64(precision, scale) => {
+ validate_decimal_precision_and_scale::<Decimal64Type>(
+ *precision, *scale,
+ )?;
+ if *scale < 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match i64::from(10).checked_pow(*scale as u32) {
+ Some(value) => {
+ ScalarValue::Decimal64(Some(-value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
DataType::Decimal128(precision, scale) => {
validate_decimal_precision_and_scale::<Decimal128Type>(
*precision, *scale,
@@ -1603,7 +1703,7 @@ impl ScalarValue {
}
_ => {
return _not_impl_err!(
- "Can't create a negative one scalar from data_type \"{datatype:?}\""
+ "Can't create a negative one scalar from data_type \"{datatype}\""
);
}
})
@@ -1622,6 +1722,38 @@ impl ScalarValue {
DataType::Float16 => ScalarValue::Float16(Some(f16::from_f32(10.0))),
DataType::Float32 => ScalarValue::Float32(Some(10.0)),
DataType::Float64 => ScalarValue::Float64(Some(10.0)),
+ DataType::Decimal32(precision, scale) => {
+ if let Err(err) = validate_decimal_precision_and_scale::<Decimal32Type>(
+ *precision, *scale,
+ ) {
+ return _internal_err!("Invalid precision and scale {err}");
+ }
+ if *scale <= 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match 10_i32.checked_pow((*scale + 1) as u32) {
+ Some(value) => {
+ ScalarValue::Decimal32(Some(value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
+ DataType::Decimal64(precision, scale) => {
+ if let Err(err) = validate_decimal_precision_and_scale::<Decimal64Type>(
+ *precision, *scale,
+ ) {
+ return _internal_err!("Invalid precision and scale {err}");
+ }
+ if *scale <= 0 {
+ return _internal_err!("Negative scale is not supported");
+ }
+ match i64::from(10).checked_pow((*scale + 1) as u32) {
+ Some(value) => {
+ ScalarValue::Decimal64(Some(value), *precision, *scale)
+ }
+ None => return _internal_err!("Unsupported scale {scale}"),
+ }
+ }
DataType::Decimal128(precision, scale) => {
if let Err(err) = validate_decimal_precision_and_scale::<Decimal128Type>(
*precision, *scale,
@@ -1656,7 +1788,7 @@ impl ScalarValue {
}
_ => {
return _not_impl_err!(
- "Can't create a ten scalar from data_type \"{datatype:?}\""
+ "Can't create a ten scalar from data_type \"{datatype}\""
);
}
})
@@ -1674,6 +1806,12 @@ impl ScalarValue {
ScalarValue::Int16(_) => DataType::Int16,
ScalarValue::Int32(_) => DataType::Int32,
ScalarValue::Int64(_) => DataType::Int64,
+ ScalarValue::Decimal32(_, precision, scale) => {
+ DataType::Decimal32(*precision, *scale)
+ }
+ ScalarValue::Decimal64(_, precision, scale) => {
+ DataType::Decimal64(*precision, *scale)
+ }
ScalarValue::Decimal128(_, precision, scale) => {
DataType::Decimal128(*precision, *scale)
}
@@ -1796,6 +1934,24 @@ impl ScalarValue {
);
Ok(ScalarValue::IntervalMonthDayNano(Some(val)))
}
+ ScalarValue::Decimal32(Some(v), precision, scale) => {
+ Ok(ScalarValue::Decimal32(
+ Some(neg_checked_with_ctx(*v, || {
+ format!("In negation of Decimal32({v}, {precision}, {scale})")
+ })?),
+ *precision,
+ *scale,
+ ))
+ }
+ ScalarValue::Decimal64(Some(v), precision, scale) => {
+ Ok(ScalarValue::Decimal64(
+ Some(neg_checked_with_ctx(*v, || {
+ format!("In negation of Decimal64({v}, {precision}, {scale})")
+ })?),
+ *precision,
+ *scale,
+ ))
+ }
ScalarValue::Decimal128(Some(v), precision, scale) => {
Ok(ScalarValue::Decimal128(
Some(neg_checked_with_ctx(*v, || {
@@ -1947,6 +2103,8 @@ impl ScalarValue {
ScalarValue::Float16(v) => v.is_none(),
ScalarValue::Float32(v) => v.is_none(),
ScalarValue::Float64(v) => v.is_none(),
+ ScalarValue::Decimal32(v, _, _) => v.is_none(),
+ ScalarValue::Decimal64(v, _, _) => v.is_none(),
ScalarValue::Decimal128(v, _, _) => v.is_none(),
ScalarValue::Decimal256(v, _, _) => v.is_none(),
ScalarValue::Int8(v) => v.is_none(),
@@ -2202,19 +2360,19 @@ impl ScalarValue {
}
let array: ArrayRef = match &data_type {
- DataType::Decimal32(_precision, _scale) => {
- return _not_impl_err!(
- "Decimal32 not supported in ScalarValue::iter_to_array"
- );
+ DataType::Decimal32(precision, scale) => {
+ let decimal_array =
+ ScalarValue::iter_to_decimal32_array(scalars, *precision, *scale)?;
+ Arc::new(decimal_array)
}
- DataType::Decimal64(_precision, _scale) => {
- return _not_impl_err!(
- "Decimal64 not supported in ScalarValue::iter_to_array"
- );
+ DataType::Decimal64(precision, scale) => {
+ let decimal_array =
+ ScalarValue::iter_to_decimal64_array(scalars, *precision, *scale)?;
+ Arc::new(decimal_array)
}
DataType::Decimal128(precision, scale) => {
let decimal_array =
- ScalarValue::iter_to_decimal_array(scalars, *precision, *scale)?;
+ ScalarValue::iter_to_decimal128_array(scalars, *precision, *scale)?;
Arc::new(decimal_array)
}
DataType::Decimal256(precision, scale) => {
@@ -2364,7 +2522,7 @@ impl ScalarValue {
DataType::UInt16 => dict_from_values::<UInt16Type>(values)?,
DataType::UInt32 => dict_from_values::<UInt32Type>(values)?,
DataType::UInt64 => dict_from_values::<UInt64Type>(values)?,
- _ => unreachable!("Invalid dictionary keys type: {:?}", key_type),
+ _ => unreachable!("Invalid dictionary keys type: {}", key_type),
}
}
DataType::FixedSizeBinary(size) => {
@@ -2375,7 +2533,7 @@ impl ScalarValue {
} else {
_exec_err!(
"Inconsistent types in ScalarValue::iter_to_array. \
- Expected {data_type:?}, got {sv:?}"
+ Expected {data_type}, got {sv:?}"
)
}
})
@@ -2423,7 +2581,43 @@ impl ScalarValue {
Ok(new_null_array(&DataType::Null, length))
}
- fn iter_to_decimal_array(
+ fn iter_to_decimal32_array(
+ scalars: impl IntoIterator<Item = ScalarValue>,
+ precision: u8,
+ scale: i8,
+ ) -> Result<Decimal32Array> {
+ let array = scalars
+ .into_iter()
+ .map(|element: ScalarValue| match element {
+ ScalarValue::Decimal32(v1, _, _) => Ok(v1),
+ s => {
+ _internal_err!("Expected ScalarValue::Null element. Received {s:?}")
+ }
+ })
+ .collect::<Result<Decimal32Array>>()?
+ .with_precision_and_scale(precision, scale)?;
+ Ok(array)
+ }
+
+ fn iter_to_decimal64_array(
+ scalars: impl IntoIterator<Item = ScalarValue>,
+ precision: u8,
+ scale: i8,
+ ) -> Result<Decimal64Array> {
+ let array = scalars
+ .into_iter()
+ .map(|element: ScalarValue| match element {
+ ScalarValue::Decimal64(v1, _, _) => Ok(v1),
+ s => {
+ _internal_err!("Expected ScalarValue::Null element. Received {s:?}")
+ }
+ })
+ .collect::<Result<Decimal64Array>>()?
+ .with_precision_and_scale(precision, scale)?;
+ Ok(array)
+ }
+
+ fn iter_to_decimal128_array(
scalars: impl IntoIterator<Item = ScalarValue>,
precision: u8,
scale: i8,
@@ -2461,7 +2655,43 @@ impl ScalarValue {
Ok(array)
}
- fn build_decimal_array(
+ fn build_decimal32_array(
+ value: Option<i32>,
+ precision: u8,
+ scale: i8,
+ size: usize,
+ ) -> Result<Decimal32Array> {
+ Ok(match value {
+ Some(val) => Decimal32Array::from(vec![val; size])
+ .with_precision_and_scale(precision, scale)?,
+ None => {
+ let mut builder = Decimal32Array::builder(size)
+ .with_precision_and_scale(precision, scale)?;
+ builder.append_nulls(size);
+ builder.finish()
+ }
+ })
+ }
+
+ fn build_decimal64_array(
+ value: Option<i64>,
+ precision: u8,
+ scale: i8,
+ size: usize,
+ ) -> Result<Decimal64Array> {
+ Ok(match value {
+ Some(val) => Decimal64Array::from(vec![val; size])
+ .with_precision_and_scale(precision, scale)?,
+ None => {
+ let mut builder = Decimal64Array::builder(size)
+ .with_precision_and_scale(precision, scale)?;
+ builder.append_nulls(size);
+ builder.finish()
+ }
+ })
+ }
+
+ fn build_decimal128_array(
value: Option<i128>,
precision: u8,
scale: i8,
@@ -2640,8 +2870,14 @@ impl ScalarValue {
/// - a `Dictionary` that fails be converted to a dictionary array of size
pub fn to_array_of_size(&self, size: usize) -> Result<ArrayRef> {
Ok(match self {
+ ScalarValue::Decimal32(e, precision, scale) => Arc::new(
+ ScalarValue::build_decimal32_array(*e, *precision, *scale, size)?,
+ ),
+ ScalarValue::Decimal64(e, precision, scale) => Arc::new(
+ ScalarValue::build_decimal64_array(*e, *precision, *scale, size)?,
+ ),
ScalarValue::Decimal128(e, precision, scale) => Arc::new(
- ScalarValue::build_decimal_array(*e, *precision, *scale, size)?,
+ ScalarValue::build_decimal128_array(*e, *precision, *scale, size)?,
),
ScalarValue::Decimal256(e, precision, scale) => Arc::new(
ScalarValue::build_decimal256_array(*e, *precision, *scale, size)?,
@@ -2937,7 +3173,7 @@ impl ScalarValue {
DataType::UInt16 => dict_from_scalar::<UInt16Type>(v, size)?,
DataType::UInt32 => dict_from_scalar::<UInt32Type>(v, size)?,
DataType::UInt64 => dict_from_scalar::<UInt64Type>(v, size)?,
- _ => unreachable!("Invalid dictionary keys type: {:?}", key_type),
+ _ => unreachable!("Invalid dictionary keys type: {}", key_type),
}
}
ScalarValue::Null => get_or_create_cached_null_array(size),
@@ -2951,6 +3187,24 @@ impl ScalarValue {
scale: i8,
) -> Result<ScalarValue> {
match array.data_type() {
+ DataType::Decimal32(_, _) => {
+ let array = as_decimal32_array(array)?;
+ if array.is_null(index) {
+ Ok(ScalarValue::Decimal32(None, precision, scale))
+ } else {
+ let value = array.value(index);
+ Ok(ScalarValue::Decimal32(Some(value), precision, scale))
+ }
+ }
+ DataType::Decimal64(_, _) => {
+ let array = as_decimal64_array(array)?;
+ if array.is_null(index) {
+ Ok(ScalarValue::Decimal64(None, precision, scale))
+ } else {
+ let value = array.value(index);
+ Ok(ScalarValue::Decimal64(Some(value), precision, scale))
+ }
+ }
DataType::Decimal128(_, _) => {
let array = as_decimal128_array(array)?;
if array.is_null(index) {
@@ -2969,7 +3223,9 @@ impl ScalarValue {
Ok(ScalarValue::Decimal256(Some(value), precision, scale))
}
}
- _ => _internal_err!("Unsupported decimal type"),
+ other => {
+ unreachable!("Invalid type isn't decimal: {other:?}")
+ }
}
}
@@ -3050,17 +3306,30 @@ impl ScalarValue {
/// assert_eq!(scalar_vec, expected);
/// ```
pub fn convert_array_to_scalar_vec(array: &dyn Array) -> Result<Vec<Vec<Self>>> {
- let mut scalars = Vec::with_capacity(array.len());
-
- for index in 0..array.len() {
- let nested_array = array.as_list::<i32>().value(index);
- let scalar_values = (0..nested_array.len())
- .map(|i| ScalarValue::try_from_array(&nested_array, i))
- .collect::<Result<Vec<_>>>()?;
- scalars.push(scalar_values);
+ fn generic_collect<OffsetSize: OffsetSizeTrait>(
+ array: &dyn Array,
+ ) -> Result<Vec<Vec<ScalarValue>>> {
+ array
+ .as_list::<OffsetSize>()
+ .iter()
+ .map(|nested_array| match nested_array {
+ Some(nested_array) => (0..nested_array.len())
+ .map(|i| ScalarValue::try_from_array(&nested_array, i))
+ .collect::<Result<Vec<_>>>(),
+ // TODO: what can we put for null?
+ // https://github.com/apache/datafusion/issues/17749
+ None => Ok(vec![]),
+ })
+ .collect()
}
- Ok(scalars)
+ match array.data_type() {
+ DataType::List(_) => generic_collect::<i32>(array),
+ DataType::LargeList(_) => generic_collect::<i64>(array),
+ _ => _internal_err!(
+ "ScalarValue::convert_array_to_scalar_vec input must be a List/LargeList type"
+ ),
+ }
}
#[deprecated(
@@ -3083,6 +3352,16 @@ impl ScalarValue {
Ok(match array.data_type() {
DataType::Null => ScalarValue::Null,
+ DataType::Decimal32(precision, scale) => {
+ ScalarValue::get_decimal_value_from_array(
+ array, index, *precision, *scale,
+ )?
+ }
+ DataType::Decimal64(precision, scale) => {
+ ScalarValue::get_decimal_value_from_array(
+ array, index, *precision, *scale,
+ )?
+ }
DataType::Decimal128(precision, scale) => {
ScalarValue::get_decimal_value_from_array(
array, index, *precision, *scale,
@@ -3197,7 +3476,7 @@ impl ScalarValue {
DataType::UInt16 => get_dict_value::<UInt16Type>(array, index)?,
DataType::UInt32 => get_dict_value::<UInt32Type>(array, index)?,
DataType::UInt64 => get_dict_value::<UInt64Type>(array, index)?,
- _ => unreachable!("Invalid dictionary keys type: {:?}", key_type),
+ _ => unreachable!("Invalid dictionary keys type: {}", key_type),
};
// look up the index in the values dictionary
let value = match values_index {
@@ -3343,6 +3622,44 @@ impl ScalarValue {
ScalarValue::try_from_array(&cast_arr, 0)
}
+ fn eq_array_decimal32(
+ array: &ArrayRef,
+ index: usize,
+ value: Option<&i32>,
+ precision: u8,
+ scale: i8,
+ ) -> Result<bool> {
+ let array = as_decimal32_array(array)?;
+ if array.precision() != precision || array.scale() != scale {
+ return Ok(false);
+ }
+ let is_null = array.is_null(index);
+ if let Some(v) = value {
+ Ok(!array.is_null(index) && array.value(index) == *v)
+ } else {
+ Ok(is_null)
+ }
+ }
+
+ fn eq_array_decimal64(
+ array: &ArrayRef,
+ index: usize,
+ value: Option<&i64>,
+ precision: u8,
+ scale: i8,
+ ) -> Result<bool> {
+ let array = as_decimal64_array(array)?;
+ if array.precision() != precision || array.scale() != scale {
+ return Ok(false);
+ }
+ let is_null = array.is_null(index);
+ if let Some(v) = value {
+ Ok(!array.is_null(index) && array.value(index) == *v)
+ } else {
+ Ok(is_null)
+ }
+ }
+
fn eq_array_decimal(
array: &ArrayRef,
index: usize,
@@ -3410,6 +3727,24 @@ impl ScalarValue {
#[inline]
pub fn eq_array(&self, array: &ArrayRef, index: usize) -> Result<bool> {
Ok(match self {
+ ScalarValue::Decimal32(v, precision, scale) => {
+ ScalarValue::eq_array_decimal32(
+ array,
+ index,
+ v.as_ref(),
+ *precision,
+ *scale,
+ )?
+ }
+ ScalarValue::Decimal64(v, precision, scale) => {
+ ScalarValue::eq_array_decimal64(
+ array,
+ index,
+ v.as_ref(),
+ *precision,
+ *scale,
+ )?
+ }
ScalarValue::Decimal128(v, precision, scale) => {
ScalarValue::eq_array_decimal(
array,
@@ -3571,7 +3906,7 @@ impl ScalarValue {
DataType::UInt16 => get_dict_value::<UInt16Type>(array, index)?,
DataType::UInt32 => get_dict_value::<UInt32Type>(array, index)?,
DataType::UInt64 => get_dict_value::<UInt64Type>(array, index)?,
- _ => unreachable!("Invalid dictionary keys type: {:?}", key_type),
+ _ => unreachable!("Invalid dictionary keys type: {}", key_type),
};
// was the value in the array non null?
match values_index {
@@ -3608,6 +3943,8 @@ impl ScalarValue {
| ScalarValue::Float16(_)
| ScalarValue::Float32(_)
| ScalarValue::Float64(_)
+ | ScalarValue::Decimal32(_, _, _)
+ | ScalarValue::Decimal64(_, _, _)
| ScalarValue::Decimal128(_, _, _)
| ScalarValue::Decimal256(_, _, _)
| ScalarValue::Int8(_)
@@ -3717,6 +4054,8 @@ impl ScalarValue {
| ScalarValue::Float16(_)
| ScalarValue::Float32(_)
| ScalarValue::Float64(_)
+ | ScalarValue::Decimal32(_, _, _)
+ | ScalarValue::Decimal64(_, _, _)
| ScalarValue::Decimal128(_, _, _)
| ScalarValue::Decimal256(_, _, _)
| ScalarValue::Int8(_)
@@ -4230,6 +4569,12 @@ macro_rules! format_option {
impl fmt::Display for ScalarValue {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
+ ScalarValue::Decimal32(v, p, s) => {
+ write!(f, "{v:?},{p:?},{s:?}")?;
+ }
+ ScalarValue::Decimal64(v, p, s) => {
+ write!(f, "{v:?},{p:?},{s:?}")?;
+ }
ScalarValue::Decimal128(v, p, s) => {
write!(f, "{v:?},{p:?},{s:?}")?;
}
@@ -4419,6 +4764,8 @@ fn fmt_binary(data: &[u8], f: &mut fmt::Formatter) -> fmt::Result {
impl fmt::Debug for ScalarValue {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
+ ScalarValue::Decimal32(_, _, _) => write!(f, "Decimal32({self})"),
+ ScalarValue::Decimal64(_, _, _) => write!(f, "Decimal64({self})"),
ScalarValue::Decimal128(_, _, _) => write!(f, "Decimal128({self})"),
ScalarValue::Decimal256(_, _, _) => write!(f, "Decimal256({self})"),
ScalarValue::Boolean(_) => write!(f, "Boolean({self})"),
@@ -4614,6 +4961,8 @@ impl ScalarType<i32> for Date32Type {
#[cfg(test)]
mod tests {
+ use std::sync::Arc;
+
use super::*;
use crate::cast::{as_list_array, as_map_array, as_struct_array};
use crate::test_util::batches_to_string;
@@ -4622,7 +4971,7 @@ mod tests {
NullArray, NullBufferBuilder, OffsetSizeTrait, PrimitiveBuilder, RecordBatch,
StringBuilder, StringDictionaryBuilder, StructBuilder, UnionBuilder,
};
- use arrow::buffer::{Buffer, OffsetBuffer};
+ use arrow::buffer::{Buffer, NullBuffer, OffsetBuffer};
use arrow::compute::{is_null, kernels};
use arrow::datatypes::{
ArrowNumericType, Fields, Float64Type, DECIMAL256_MAX_PRECISION,
@@ -8663,4 +9012,66 @@ mod tests {
_ => panic!("Expected TimestampMillisecond with timezone"),
}
}
+
+ #[test]
+ fn test_convert_array_to_scalar_vec() {
+ // Regular ListArray
+ let list = ListArray::from_iter_primitive::<Int64Type, _, _>(vec![
+ Some(vec![Some(1), Some(2)]),
+ None,
+ Some(vec![Some(3), None, Some(4)]),
+ ]);
+ let converted = ScalarValue::convert_array_to_scalar_vec(&list).unwrap();
+ assert_eq!(
+ converted,
+ vec![
+ vec![ScalarValue::Int64(Some(1)), ScalarValue::Int64(Some(2))],
+ vec![],
+ vec![
+ ScalarValue::Int64(Some(3)),
+ ScalarValue::Int64(None),
+ ScalarValue::Int64(Some(4))
+ ],
+ ]
+ );
+
+ // Regular LargeListArray
+ let large_list = LargeListArray::from_iter_primitive::<Int64Type, _, _>(vec![
+ Some(vec![Some(1), Some(2)]),
+ None,
+ Some(vec![Some(3), None, Some(4)]),
+ ]);
+ let converted = ScalarValue::convert_array_to_scalar_vec(&large_list).unwrap();
+ assert_eq!(
+ converted,
+ vec![
+ vec![ScalarValue::Int64(Some(1)), ScalarValue::Int64(Some(2))],
+ vec![],
+ vec![
+ ScalarValue::Int64(Some(3)),
+ ScalarValue::Int64(None),
+ ScalarValue::Int64(Some(4))
+ ],
+ ]
+ );
+
+ // Funky (null slot has non-zero list offsets)
+ // Offsets + Values looks like this: [[1, 2], [3, 4], [5]]
+ // But with NullBuffer it's like this: [[1, 2], NULL, [5]]
+ let funky = ListArray::new(
+ Field::new_list_field(DataType::Int64, true).into(),
+ OffsetBuffer::new(vec![0, 2, 4, 5].into()),
+ Arc::new(Int64Array::from(vec![1, 2, 3, 4, 5, 6])),
+ Some(NullBuffer::from(vec![true, false, true])),
+ );
+ let converted = ScalarValue::convert_array_to_scalar_vec(&funky).unwrap();
+ assert_eq!(
+ converted,
+ vec![
+ vec![ScalarValue::Int64(Some(1)), ScalarValue::Int64(Some(2))],
+ vec![],
+ vec![ScalarValue::Int64(Some(5))],
+ ]
+ );
+ }
}
diff --git a/datafusion/common/src/types/native.rs b/datafusion/common/src/types/native.rs
index 76629e555..5cef0adfb 100644
--- a/datafusion/common/src/types/native.rs
+++ b/datafusion/common/src/types/native.rs
@@ -23,6 +23,7 @@ use crate::error::{Result, _internal_err};
use arrow::compute::can_cast_types;
use arrow::datatypes::{
DataType, Field, FieldRef, Fields, IntervalUnit, TimeUnit, UnionFields,
+ DECIMAL128_MAX_PRECISION, DECIMAL32_MAX_PRECISION, DECIMAL64_MAX_PRECISION,
};
use std::{fmt::Display, sync::Arc};
@@ -185,7 +186,7 @@ pub enum NativeType {
impl Display for NativeType {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
- write!(f, "NativeType::{self:?}")
+ write!(f, "{self:?}") // TODO: nicer formatting
}
}
@@ -228,7 +229,15 @@ impl LogicalType for NativeType {
(Self::Float16, _) => Float16,
(Self::Float32, _) => Float32,
(Self::Float64, _) => Float64,
- (Self::Decimal(p, s), _) if p <= &38 => Decimal128(*p, *s),
+ (Self::Decimal(p, s), _) if *p <= DECIMAL32_MAX_PRECISION => {
+ Decimal32(*p, *s)
+ }
+ (Self::Decimal(p, s), _) if *p <= DECIMAL64_MAX_PRECISION => {
+ Decimal64(*p, *s)
+ }
+ (Self::Decimal(p, s), _) if *p <= DECIMAL128_MAX_PRECISION => {
+ Decimal128(*p, *s)
+ }
(Self::Decimal(p, s), _) => Decimal256(*p, *s),
(Self::Timestamp(tu, tz), _) => Timestamp(*tu, tz.clone()),
// If given type is Date, return the same type
@@ -352,10 +361,10 @@ impl LogicalType for NativeType {
}
_ => {
return _internal_err!(
- "Unavailable default cast for native type {:?} from physical type {:?}",
- self,
- origin
- )
+ "Unavailable default cast for native type {} from physical type {}",
+ self,
+ origin
+ )
}
})
}
@@ -472,4 +481,9 @@ impl NativeType {
pub fn is_duration(&self) -> bool {
matches!(self, NativeType::Duration(_))
}
+
+ #[inline]
+ pub fn is_binary(&self) -> bool {
+ matches!(self, NativeType::Binary | NativeType::FixedSizeBinary(_))
+ }
}
diff --git a/datafusion/core/Cargo.toml b/datafusion/core/Cargo.toml
index 620885697..1e5d1d104 100644
--- a/datafusion/core/Cargo.toml
+++ b/datafusion/core/Cargo.toml
@@ -149,6 +149,7 @@ parking_lot = { workspace = true }
parquet = { workspace = true, optional = true, default-features = true }
rand = { workspace = true }
regex = { workspace = true }
+rstest = { workspace = true }
serde = { version = "1.0", default-features = false, features = ["derive"], optional = true }
sqlparser = { workspace = true, optional = true }
tempfile = { workspace = true }
@@ -177,7 +178,7 @@ rand_distr = "0.5"
regex = { workspace = true }
rstest = { workspace = true }
serde_json = { workspace = true }
-sysinfo = "0.37.0"
+sysinfo = "0.37.1"
test-utils = { path = "../../test-utils" }
tokio = { workspace = true, features = ["rt-multi-thread", "parking_lot", "fs"] }
diff --git a/datafusion/core/README.md b/datafusion/core/README.md
index b5501087d..859fcb9c0 100644
--- a/datafusion/core/README.md
+++ b/datafusion/core/README.md
@@ -17,15 +17,12 @@
under the License.
-->
-# DataFusion Core
+<!--
+ Note the main crates.io landing page https://crates.io/crates/datafusion
+ uses the workspace README.md file, not this file
+-->
-DataFusion is an extensible query execution framework, written in Rust,
-that uses Apache Arrow as its in-memory format.
+# Apache DataFusion Core
This crate contains the main entry points and high level DataFusion APIs such as
`SessionContext`, `DataFrame` and `ListingTable`.
-
-For more information, please see:
-
-- [DataFusion Website](https://datafusion.apache.org)
-- [DataFusion API Docs](https://docs.rs/datafusion/latest/datafusion/)
diff --git a/datafusion/core/benches/sql_planner.rs b/datafusion/core/benches/sql_planner.rs
index c71191507..3be8668b2 100644
--- a/datafusion/core/benches/sql_planner.rs
+++ b/datafusion/core/benches/sql_planner.rs
@@ -476,7 +476,8 @@ fn criterion_benchmark(c: &mut Criterion) {
});
});
- for partitioning_columns in [4, 7, 8] {
+ // It was observed in production that queries with window functions sometimes partition over more than 30 columns
+ for partitioning_columns in [4, 7, 8, 12, 30] {
c.bench_function(
&format!(
"physical_window_function_partition_by_{partitioning_columns}_on_values"
@@ -663,6 +664,9 @@ fn criterion_benchmark(c: &mut Criterion) {
};
let raw_tpcds_sql_queries = (1..100)
+ // skip query 75 until it is fixed
+ // https://github.com/apache/datafusion/issues/17801
+ .filter(|q| *q != 75)
.map(|q| std::fs::read_to_string(format!("{tests_path}tpc-ds/{q}.sql")).unwrap())
.collect::<Vec<_>>();
diff --git a/datafusion/core/src/dataframe/mod.rs b/datafusion/core/src/dataframe/mod.rs
index 9832c0e9d..02c2c81ad 100644
--- a/datafusion/core/src/dataframe/mod.rs
+++ b/datafusion/core/src/dataframe/mod.rs
@@ -43,7 +43,7 @@ use crate::physical_plan::{
use crate::prelude::SessionContext;
use std::any::Any;
use std::borrow::Cow;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use arrow::array::{Array, ArrayRef, Int64Array, StringArray};
@@ -2023,31 +2023,38 @@ impl DataFrame {
pub fn with_column(self, name: &str, expr: Expr) -> Result<DataFrame> {
let window_func_exprs = find_window_exprs([&expr]);
- let (window_fn_str, plan) = if window_func_exprs.is_empty() {
- (None, self.plan)
+ let original_names: HashSet<String> = self
+ .plan
+ .schema()
+ .iter()
+ .map(|(_, f)| f.name().clone())
+ .collect();
+
+ // Maybe build window plan
+ let plan = if window_func_exprs.is_empty() {
+ self.plan
} else {
- (
- Some(window_func_exprs[0].to_string()),
- LogicalPlanBuilder::window_plan(self.plan, window_func_exprs)?,
- )
+ LogicalPlanBuilder::window_plan(self.plan, window_func_exprs)?
};
- let mut col_exists = false;
let new_column = expr.alias(name);
+ let mut col_exists = false;
+
let mut fields: Vec<(Expr, bool)> = plan
.schema()
.iter()
.filter_map(|(qualifier, field)| {
+ // Skip new fields introduced by window_plan
+ if !original_names.contains(field.name()) {
+ return None;
+ }
+
if field.name() == name {
col_exists = true;
Some((new_column.clone(), true))
} else {
let e = col(Column::from((qualifier, field)));
- window_fn_str
- .as_ref()
- .filter(|s| *s == &e.to_string())
- .is_none()
- .then_some((e, self.projection_requires_validation))
+ Some((e, self.projection_requires_validation))
}
})
.collect();
@@ -2440,8 +2447,7 @@ impl TableProvider for DataFrameTableProvider {
}
fn schema(&self) -> SchemaRef {
- let schema: Schema = self.plan.schema().as_ref().into();
- Arc::new(schema)
+ Arc::clone(self.plan.schema().inner())
}
fn table_type(&self) -> TableType {
diff --git a/datafusion/core/src/dataframe/parquet.rs b/datafusion/core/src/dataframe/parquet.rs
index 735c69af7..6fb00df34 100644
--- a/datafusion/core/src/dataframe/parquet.rs
+++ b/datafusion/core/src/dataframe/parquet.rs
@@ -102,6 +102,7 @@ impl DataFrame {
#[cfg(test)]
mod tests {
+ use rstest::rstest;
use std::collections::HashMap;
use std::sync::Arc;
@@ -247,9 +248,12 @@ mod tests {
Ok(())
}
+ #[rstest]
#[cfg(feature = "parquet_encryption")]
#[tokio::test]
- async fn roundtrip_parquet_with_encryption() -> Result<()> {
+ async fn roundtrip_parquet_with_encryption(
+ #[values(false, true)] allow_single_file_parallelism: bool,
+ ) -> Result<()> {
use parquet::encryption::decrypt::FileDecryptionProperties;
use parquet::encryption::encrypt::FileEncryptionProperties;
@@ -278,6 +282,7 @@ mod tests {
// Write encrypted parquet using write_parquet
let mut options = TableParquetOptions::default();
options.crypto.file_encryption = Some((&encrypt).into());
+ options.global.allow_single_file_parallelism = allow_single_file_parallelism;
df.write_parquet(
tempfile_str.as_str(),
diff --git a/datafusion/core/src/datasource/file_format/csv.rs b/datafusion/core/src/datasource/file_format/csv.rs
index 6c4897f71..edbbea97a 100644
--- a/datafusion/core/src/datasource/file_format/csv.rs
+++ b/datafusion/core/src/datasource/file_format/csv.rs
@@ -1233,7 +1233,7 @@ mod tests {
fn csv_values(line_number: usize) -> (i32, f64, bool, String) {
let int_value = line_number as i32;
let float_value = line_number as f64;
- let bool_value = line_number % 2 == 0;
+ let bool_value = line_number.is_multiple_of(2);
let char_value = format!("{line_number}-string");
(int_value, float_value, bool_value, char_value)
}
diff --git a/datafusion/core/src/datasource/listing_table_factory.rs b/datafusion/core/src/datasource/listing_table_factory.rs
index 218a1fedb..f98297d0e 100644
--- a/datafusion/core/src/datasource/listing_table_factory.rs
+++ b/datafusion/core/src/datasource/listing_table_factory.rs
@@ -27,7 +27,7 @@ use crate::datasource::listing::{
};
use crate::execution::context::SessionState;
-use arrow::datatypes::{DataType, SchemaRef};
+use arrow::datatypes::DataType;
use datafusion_common::{arrow_datafusion_err, plan_err, DataFusionError, ToDFSchema};
use datafusion_common::{config_datafusion_err, Result};
use datafusion_expr::CreateExternalTable;
@@ -105,7 +105,7 @@ impl TableProviderFactory for ListingTableFactory {
.collect::<Vec<_>>(),
)
} else {
- let schema: SchemaRef = Arc::new(cmd.schema.as_ref().to_owned().into());
+ let schema = Arc::clone(cmd.schema.inner());
let table_partition_cols = cmd
.table_partition_cols
.iter()
@@ -238,6 +238,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -278,6 +279,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -322,6 +324,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -373,6 +376,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -416,6 +420,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -455,6 +460,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
@@ -495,6 +501,7 @@ mod tests {
schema: Arc::new(DFSchema::empty()),
table_partition_cols: vec![],
if_not_exists: false,
+ or_replace: false,
temporary: false,
definition: None,
order_exprs: vec![],
diff --git a/datafusion/core/src/datasource/physical_plan/parquet.rs b/datafusion/core/src/datasource/physical_plan/parquet.rs
index 6f1c6c417..7c9767cee 100644
--- a/datafusion/core/src/datasource/physical_plan/parquet.rs
+++ b/datafusion/core/src/datasource/physical_plan/parquet.rs
@@ -2018,14 +2018,14 @@ mod tests {
let out_dir = tmp_dir.as_ref().to_str().unwrap().to_string() + "/out";
fs::create_dir(&out_dir).unwrap();
let df = ctx.sql("SELECT c1, c2 FROM test").await?;
- let schema: Schema = df.schema().into();
+ let schema = Arc::clone(df.schema().inner());
// Register a listing table - this will use all files in the directory as data sources
// for the query
ctx.register_listing_table(
"my_table",
&out_dir,
listing_options,
- Some(Arc::new(schema)),
+ Some(schema),
None,
)
.await
diff --git a/datafusion/core/src/execution/context/mod.rs b/datafusion/core/src/execution/context/mod.rs
index e7a66c4f9..011b84aa5 100644
--- a/datafusion/core/src/execution/context/mod.rs
+++ b/datafusion/core/src/execution/context/mod.rs
@@ -792,19 +792,44 @@ impl SessionContext {
return not_impl_err!("Temporary tables not supported");
}
- if exist {
- match cmd.if_not_exists {
- true => return self.return_empty_dataframe(),
- false => {
- return exec_err!("Table '{}' already exists", cmd.name);
+ match (cmd.if_not_exists, cmd.or_replace, exist) {
+ (true, false, true) => self.return_empty_dataframe(),
+ (false, true, true) => {
+ let result = self
+ .find_and_deregister(cmd.name.clone(), TableType::Base)
+ .await;
+
+ match result {
+ Ok(true) => {
+ let table_provider: Arc<dyn TableProvider> =
+ self.create_custom_table(cmd).await?;
+ self.register_table(cmd.name.clone(), table_provider)?;
+ self.return_empty_dataframe()
+ }
+ Ok(false) => {
+ let table_provider: Arc<dyn TableProvider> =
+ self.create_custom_table(cmd).await?;
+ self.register_table(cmd.name.clone(), table_provider)?;
+ self.return_empty_dataframe()
+ }
+ Err(e) => {
+ exec_err!("Errored while deregistering external table: {}", e)
+ }
}
}
+ (true, true, true) => {
+ exec_err!("'IF NOT EXISTS' cannot coexist with 'REPLACE'")
+ }
+ (_, _, false) => {
+ let table_provider: Arc<dyn TableProvider> =
+ self.create_custom_table(cmd).await?;
+ self.register_table(cmd.name.clone(), table_provider)?;
+ self.return_empty_dataframe()
+ }
+ (false, false, true) => {
+ exec_err!("External table '{}' already exists", cmd.name)
+ }
}
-
- let table_provider: Arc<dyn TableProvider> =
- self.create_custom_table(cmd).await?;
- self.register_table(cmd.name.clone(), table_provider)?;
- self.return_empty_dataframe()
}
async fn create_memory_table(&self, cmd: CreateMemoryTable) -> Result<DataFrame> {
@@ -830,7 +855,7 @@ impl SessionContext {
(true, false, Ok(_)) => self.return_empty_dataframe(),
(false, true, Ok(_)) => {
self.deregister_table(name.clone())?;
- let schema = Arc::new(input.schema().as_ref().into());
+ let schema = Arc::clone(input.schema().inner());
let physical = DataFrame::new(self.state(), input);
let batches: Vec<_> = physical.collect_partitioned().await?;
@@ -848,8 +873,7 @@ impl SessionContext {
exec_err!("'IF NOT EXISTS' cannot coexist with 'REPLACE'")
}
(_, _, Err(_)) => {
- let df_schema = input.schema();
- let schema = Arc::new(df_schema.as_ref().into());
+ let schema = Arc::clone(input.schema().inner());
let physical = DataFrame::new(self.state(), input);
let batches: Vec<_> = physical.collect_partitioned().await?;
@@ -1730,6 +1754,14 @@ impl FunctionRegistry for SessionContext {
) -> Result<()> {
self.state.write().register_expr_planner(expr_planner)
}
+
+ fn udafs(&self) -> HashSet<String> {
+ self.state.read().udafs()
+ }
+
+ fn udwfs(&self) -> HashSet<String> {
+ self.state.read().udwfs()
+ }
}
/// Create a new task context instance from SessionContext
diff --git a/datafusion/core/src/execution/session_state.rs b/datafusion/core/src/execution/session_state.rs
index f65829090..b04004dd4 100644
--- a/datafusion/core/src/execution/session_state.rs
+++ b/datafusion/core/src/execution/session_state.rs
@@ -1917,6 +1917,14 @@ impl FunctionRegistry for SessionState {
self.expr_planners.push(expr_planner);
Ok(())
}
+
+ fn udafs(&self) -> HashSet<String> {
+ self.aggregate_functions.keys().cloned().collect()
+ }
+
+ fn udwfs(&self) -> HashSet<String> {
+ self.window_functions.keys().cloned().collect()
+ }
}
impl OptimizerConfig for SessionState {
diff --git a/datafusion/core/src/physical_planner.rs b/datafusion/core/src/physical_planner.rs
index 15d325288..6b4d2592f 100644
--- a/datafusion/core/src/physical_planner.rs
+++ b/datafusion/core/src/physical_planner.rs
@@ -59,12 +59,10 @@ use crate::schema_equivalence::schema_satisfied_by;
use arrow::array::{builder::StringBuilder, RecordBatch};
use arrow::compute::SortOptions;
-use arrow::datatypes::{Schema, SchemaRef};
+use arrow::datatypes::Schema;
use datafusion_catalog::ScanArgs;
use datafusion_common::display::ToStringifiedPlan;
-use datafusion_common::tree_node::{
- Transformed, TransformedResult, TreeNode, TreeNodeRecursion, TreeNodeVisitor,
-};
+use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion, TreeNodeVisitor};
use datafusion_common::TableReference;
use datafusion_common::{
exec_err, internal_datafusion_err, internal_err, not_impl_err, plan_err, DFSchema,
@@ -85,7 +83,7 @@ use datafusion_expr::{
WindowFrameBound, WriteOp,
};
use datafusion_physical_expr::aggregate::{AggregateExprBuilder, AggregateFunctionExpr};
-use datafusion_physical_expr::expressions::{Column, Literal};
+use datafusion_physical_expr::expressions::Literal;
use datafusion_physical_expr::{
create_physical_sort_exprs, LexOrdering, PhysicalSortExpr,
};
@@ -468,7 +466,6 @@ impl DefaultPhysicalPlanner {
Arc::clone(res.plan())
}
LogicalPlan::Values(Values { values, schema }) => {
- let exec_schema = schema.as_ref().to_owned().into();
let exprs = values
.iter()
.map(|row| {
@@ -479,27 +476,23 @@ impl DefaultPhysicalPlanner {
.collect::<Result<Vec<Arc<dyn PhysicalExpr>>>>()
})
.collect::<Result<Vec<_>>>()?;
- MemorySourceConfig::try_new_as_values(SchemaRef::new(exec_schema), exprs)?
+ MemorySourceConfig::try_new_as_values(Arc::clone(schema.inner()), exprs)?
as _
}
LogicalPlan::EmptyRelation(EmptyRelation {
produce_one_row: false,
schema,
- }) => Arc::new(EmptyExec::new(SchemaRef::new(
- schema.as_ref().to_owned().into(),
- ))),
+ }) => Arc::new(EmptyExec::new(Arc::clone(schema.inner()))),
LogicalPlan::EmptyRelation(EmptyRelation {
produce_one_row: true,
schema,
- }) => Arc::new(PlaceholderRowExec::new(SchemaRef::new(
- schema.as_ref().to_owned().into(),
- ))),
+ }) => Arc::new(PlaceholderRowExec::new(Arc::clone(schema.inner()))),
LogicalPlan::DescribeTable(DescribeTable {
schema,
output_schema,
}) => {
- let output_schema: Schema = output_schema.as_ref().into();
- self.plan_describe(Arc::clone(schema), Arc::new(output_schema))?
+ let output_schema = Arc::clone(output_schema.inner());
+ self.plan_describe(Arc::clone(schema), output_schema)?
}
// 1 Child
@@ -516,7 +509,7 @@ impl DefaultPhysicalPlanner {
let parsed_url = ListingTableUrl::parse(output_url)?;
let object_store_url = parsed_url.object_store();
- let schema: Schema = (**input.schema()).clone().into();
+ let schema = Arc::clone(input.schema().inner());
// Note: the DataType passed here is ignored for the purposes of writing and inferred instead
// from the schema of the RecordBatch being written. This allows COPY statements to specify only
@@ -553,7 +546,7 @@ impl DefaultPhysicalPlanner {
object_store_url,
table_paths: vec![parsed_url],
file_group: FileGroup::default(),
- output_schema: Arc::new(schema),
+ output_schema: schema,
table_partition_cols,
insert_op: InsertOp::Append,
keep_partition_by_columns,
@@ -741,9 +734,54 @@ impl DefaultPhysicalPlanner {
})
.collect::<Result<Vec<_>>>()?;
- let (aggregates, filters, _order_bys): (Vec<_>, Vec<_>, Vec<_>) =
+ let (mut aggregates, filters, _order_bys): (Vec<_>, Vec<_>, Vec<_>) =
multiunzip(agg_filter);
+ let mut async_exprs = Vec::new();
+ let num_input_columns = physical_input_schema.fields().len();
+
+ for agg_func in &mut aggregates {
+ match self.try_plan_async_exprs(
+ num_input_columns,
+ PlannedExprResult::Expr(agg_func.expressions()),
+ physical_input_schema.as_ref(),
+ )? {
+ PlanAsyncExpr::Async(
+ async_map,
+ PlannedExprResult::Expr(physical_exprs),
+ ) => {
+ async_exprs.extend(async_map.async_exprs);
+
+ if let Some(new_agg_func) = agg_func.with_new_expressions(
+ physical_exprs,
+ agg_func
+ .order_bys()
+ .iter()
+ .cloned()
+ .map(|x| x.expr)
+ .collect(),
+ ) {
+ *agg_func = Arc::new(new_agg_func);
+ } else {
+ return internal_err!("Failed to plan async expression");
+ }
+ }
+ PlanAsyncExpr::Sync(PlannedExprResult::Expr(_)) => {
+ // Do nothing
+ }
+ _ => {
+ return internal_err!(
+ "Unexpected result from try_plan_async_exprs"
+ )
+ }
+ }
+ }
+ let input_exec = if !async_exprs.is_empty() {
+ Arc::new(AsyncFuncExec::try_new(async_exprs, input_exec)?)
+ } else {
+ input_exec
+ };
+
let initial_aggr = Arc::new(AggregateExec::try_new(
AggregateMode::Partial,
groups.clone(),
@@ -933,7 +971,7 @@ impl DefaultPhysicalPlanner {
..
}) => {
let input = children.one()?;
- let schema = SchemaRef::new(schema.as_ref().to_owned().into());
+ let schema = Arc::clone(schema.inner());
let list_column_indices = list_type_columns
.iter()
.map(|(index, unnesting)| ListUnnest {
@@ -1641,7 +1679,7 @@ pub fn create_window_expr_with_name(
execution_props: &ExecutionProps,
) -> Result<Arc<dyn WindowExpr>> {
let name = name.into();
- let physical_schema: &Schema = &logical_schema.into();
+ let physical_schema = Arc::clone(logical_schema.inner());
match e {
Expr::WindowFunction(window_fun) => {
let WindowFunction {
@@ -2033,7 +2071,7 @@ impl DefaultPhysicalPlanner {
session_state: &SessionState,
) -> Result<Arc<dyn ExecutionPlan>> {
let input = self.create_physical_plan(&a.input, session_state).await?;
- let schema = SchemaRef::new((*a.schema).clone().into());
+ let schema = Arc::clone(a.schema.inner());
let show_statistics = session_state.config_options().explain.show_statistics;
Ok(Arc::new(AnalyzeExec::new(
a.verbose,
@@ -2097,7 +2135,15 @@ impl DefaultPhysicalPlanner {
"Optimized physical plan:\n{}\n",
displayable(new_plan.as_ref()).indent(false)
);
- debug!("Detailed optimized physical plan:\n{new_plan:?}");
+
+ // Don't print new_plan directly, as that may overflow the stack.
+ // For example:
+ // thread 'tokio-runtime-worker' has overflowed its stack
+ // fatal runtime error: stack overflow, aborting
+ debug!(
+ "Detailed optimized physical plan:\n{}\n",
+ displayable(new_plan.as_ref()).indent(true)
+ );
Ok(new_plan)
}
@@ -2115,7 +2161,7 @@ impl DefaultPhysicalPlanner {
// "System supplied type" --> Use debug format of the datatype
let data_type = field.data_type();
- data_types.append_value(format!("{data_type:?}"));
+ data_types.append_value(format!("{data_type}"));
// "YES if the column is possibly nullable, NO if it is known not nullable. "
let nullable_str = if field.is_nullable() { "YES" } else { "NO" };
@@ -2181,11 +2227,7 @@ impl DefaultPhysicalPlanner {
let physical_expr =
self.create_physical_expr(e, input_logical_schema, session_state);
- // Check for possible column name mismatches
- let final_physical_expr =
- maybe_fix_physical_column_name(physical_expr, &input_physical_schema);
-
- tuple_err((final_physical_expr, physical_name))
+ tuple_err((physical_expr, physical_name))
})
.collect::<Result<Vec<_>>>()?;
@@ -2272,11 +2314,13 @@ impl DefaultPhysicalPlanner {
}
}
+#[derive(Debug)]
enum PlannedExprResult {
ExprWithName(Vec<(Arc<dyn PhysicalExpr>, String)>),
Expr(Vec<Arc<dyn PhysicalExpr>>),
}
+#[derive(Debug)]
enum PlanAsyncExpr {
Sync(PlannedExprResult),
Async(AsyncMapper, PlannedExprResult),
@@ -2291,47 +2335,6 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {
}
}
-// Handle the case where the name of a physical column expression does not match the corresponding physical input fields names.
-// Physical column names are derived from the physical schema, whereas physical column expressions are derived from the logical column names.
-//
-// This is a special case that applies only to column expressions. Logical plans may slightly modify column names by appending a suffix (e.g., using ':'),
-// to avoid duplicates—since DFSchemas do not allow duplicate names. For example: `count(Int64(1)):1`.
-fn maybe_fix_physical_column_name(
- expr: Result<Arc<dyn PhysicalExpr>>,
- input_physical_schema: &SchemaRef,
-) -> Result<Arc<dyn PhysicalExpr>> {
- let Ok(expr) = expr else { return expr };
- expr.transform_down(|node| {
- if let Some(column) = node.as_any().downcast_ref::<Column>() {
- let idx = column.index();
- let physical_field = input_physical_schema.field(idx);
- let expr_col_name = column.name();
- let physical_name = physical_field.name();
-
- if expr_col_name != physical_name {
- // handle edge cases where the physical_name contains ':'.
- let colon_count = physical_name.matches(':').count();
- let mut splits = expr_col_name.match_indices(':');
- let split_pos = splits.nth(colon_count);
-
- if let Some((i, _)) = split_pos {
- let base_name = &expr_col_name[..i];
- if base_name == physical_name {
- let updated_column = Column::new(physical_name, idx);
- return Ok(Transformed::yes(Arc::new(updated_column)));
- }
- }
- }
-
- // If names already match or fix is not possible, just leave it as it is
- Ok(Transformed::no(node))
- } else {
- Ok(Transformed::no(node))
- }
- })
- .data()
-}
-
struct OptimizationInvariantChecker<'a> {
rule: &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,
}
@@ -2429,18 +2432,17 @@ mod tests {
use crate::execution::session_state::SessionStateBuilder;
use arrow::array::{ArrayRef, DictionaryArray, Int32Array};
use arrow::datatypes::{DataType, Field, Int32Type};
+ use arrow_schema::SchemaRef;
use datafusion_common::config::ConfigOptions;
use datafusion_common::{
assert_contains, DFSchemaRef, TableReference, ToDFSchema as _,
};
use datafusion_execution::runtime_env::RuntimeEnv;
use datafusion_execution::TaskContext;
- use datafusion_expr::{
- col, lit, LogicalPlanBuilder, Operator, UserDefinedLogicalNodeCore,
- };
+ use datafusion_expr::builder::subquery_alias;
+ use datafusion_expr::{col, lit, LogicalPlanBuilder, UserDefinedLogicalNodeCore};
use datafusion_functions_aggregate::count::count_all;
use datafusion_functions_aggregate::expr_fn::sum;
- use datafusion_physical_expr::expressions::{BinaryExpr, IsNotNullExpr};
use datafusion_physical_expr::EquivalenceProperties;
use datafusion_physical_plan::execution_plan::{Boundedness, EmissionType};
@@ -2742,7 +2744,7 @@ mod tests {
assert_contains!(
&e,
- r#"Error during planning: Can not find compatible types to compare Boolean with [Struct([Field { name: "foo", data_type: Boolean, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }]), Utf8]"#
+ r#"Error during planning: Can not find compatible types to compare Boolean with [Struct(foo Boolean), Utf8]"#
);
Ok(())
@@ -3001,71 +3003,6 @@ mod tests {
}
}
- #[tokio::test]
- async fn test_maybe_fix_colon_in_physical_name() {
- // The physical schema has a field name with a colon
- let schema = Schema::new(vec![Field::new("metric:avg", DataType::Int32, false)]);
- let schema_ref: SchemaRef = Arc::new(schema);
-
- // What might happen after deduplication
- let logical_col_name = "metric:avg:1";
- let expr_with_suffix =
- Arc::new(Column::new(logical_col_name, 0)) as Arc<dyn PhysicalExpr>;
- let expr_result = Ok(expr_with_suffix);
-
- // Call function under test
- let fixed_expr =
- maybe_fix_physical_column_name(expr_result, &schema_ref).unwrap();
... 21730 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org