You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by al...@apache.org on 2024/10/08 16:28:53 UTC
(datafusion) branch logical-types updated: [logical-types] update working branch (#12812)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch logical-types
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/logical-types by this push:
new f475a0f9aa [logical-types] update working branch (#12812)
f475a0f9aa is described below
commit f475a0f9aa163857cd8e92aaffd9ba16f85dff88
Author: Filippo Rossi <fi...@datadoghq.com>
AuthorDate: Tue Oct 8 18:28:49 2024 +0200
[logical-types] update working branch (#12812)
* Add support for external tables with qualified names (#12645)
* Make support schemas
* Set default name to table
* Remove print statements and stale comment
* Add tests for create table
* Fix typo
* Update datafusion/sql/src/statement.rs
Co-authored-by: Jonah Gao <jo...@msn.com>
* convert create_external_table to objectname
* Add sqllogic tests
* Fix failing tests
---------
Co-authored-by: Jonah Gao <jo...@msn.com>
* Fix Regex signature types (#12690)
* Fix Regex signature types
* Uncomment the shared tests in string_query.slt.part and removed tests copies everywhere else
* Test `LIKE` and `MATCH` with flags; Remove new tests from regexp.slt
* Refactor `ByteGroupValueBuilder` to use `MaybeNullBufferBuilder` (#12681)
* Fix malformed hex string literal in docs (#12708)
* Simplify match patterns in coercion rules (#12711)
Remove conditions where unnecessary.
Refactor to improve readability.
* Remove aggregate functions dependency on frontend (#12715)
* Remove aggregate functions dependency on frontend
DataFusion is a SQL query engine and also a reusable library for
building query engines. The core functionality should not depend on
frontend related functionalities like `sqlparser` or `datafusion-sql`.
* Remove duplicate license header
* Minor: Remove clone in `transform_to_states` (#12707)
* rm clone
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fmt
Signed-off-by: jayzhan211 <ja...@gmail.com>
---------
Signed-off-by: jayzhan211 <ja...@gmail.com>
* Refactor tests for union sorting properties, add tests for unions and constants (#12702)
* Refactor tests for union sorting properties
* update doc test
* Undo import reordering
* remove unecessary static lifetimes
* Fix: support Qualified Wildcard in count aggregate function (#12673)
* Reduce code duplication in `PrimitiveGroupValueBuilder` with const generics (#12703)
* Reduce code duplication in `PrimitiveGroupValueBuilder` with const generics
* Fix docs
* Disallow duplicated qualified field names (#12608)
* Disallow duplicated qualified field names
* Fix tests
* Optimize base64/hex decoding by pre-allocating output buffers (~2x faster) (#12675)
* add bench
* replace macro with generic function
* remove duplicated code
* optimize base64/hex decode
* Allow DynamicFileCatalog support to query partitioned file (#12683)
* support to query partitioned table for dynamic file catalog
* cargo clippy
* split partitions inferring to another function
* Support `LIMIT` Push-down logical plan optimization for `Extension` nodes (#12685)
* Update trait `UserDefinedLogicalNodeCore`
Signed-off-by: Austin Liu <au...@gmail.com>
* Update corresponding interface
Signed-off-by: Austin Liu <au...@gmail.com>
Add rewrite rule for `push-down-limit` for `Extension`
Signed-off-by: Austin Liu <au...@gmail.com>
* Add rewrite rule for `push-down-limit` for `Extension` and tests
Signed-off-by: Austin Liu <au...@gmail.com>
* Update corresponding interface
Signed-off-by: Austin Liu <au...@gmail.com>
* Reorganize to match guard
Signed-off-by: Austin Liu <au...@gmail.com>
* Clena up
Signed-off-by: Austin Liu <au...@gmail.com>
Clean up
Signed-off-by: Austin Liu <au...@gmail.com>
---------
Signed-off-by: Austin Liu <au...@gmail.com>
* Fix AvroReader: Add union resolving for nested struct arrays (#12686)
* Add union resolving for nested struct arrays
* Add test
* Change test
* Reproduce index error
* fmt
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Adds macros for creating `WindowUDF` and `WindowFunction` expression (#12693)
* Adds macro for udwf singleton
* Adds a doc comment parameter to macro
* Add doc comment for `create_udwf` macro
* Uses default constructor
* Update `Cargo.lock` in `datafusion-cli`
* Fixes: expand `$FN_NAME` in doc strings
* Adds example for macro usage
* Renames macro
* Improve doc comments
* Rename udwf macro
* Minor: doc copy edits
* Adds macro for creating fluent-style expression API
* Adds support for 1 or more parameters in expression function
* Rewrite doc comments
* Rename parameters
* Minor: formatting
* Adds doc comment for `create_udwf_expr` macro
* Improve example docs
* Hides extraneous code in doc comments
* Add a one-line readme
* Adds doc test assertions + minor formatting fixes
* Adds common macro for defining user-defined window functions
* Adds doc comment for `define_udwf_and_expr`
* Defines `RowNumber` using common macro
* Add usage example for common macro
* Adds usage for custom constructor
* Add examples for remaining patterns
* Improve doc comments for usage examples
* Rewrite inner line docs
* Rewrite `create_udwf_expr!` doc comments
* Minor doc improvements
* Fix doc test and usage example
* Add inline comments for macro patterns
* Minor: change doc comment in example
* Support unparsing plans with both Aggregation and Window functions (#12705)
* Support unparsing plans with both Aggregation and Window functions (#35)
* Fix unparsing for aggregation grouping sets
* Add test for grouping set unparsing
* Update datafusion/sql/src/unparser/utils.rs
Co-authored-by: Jax Liu <li...@gmail.com>
* Update datafusion/sql/src/unparser/utils.rs
Co-authored-by: Jax Liu <li...@gmail.com>
* Update
* More tests
---------
Co-authored-by: Jax Liu <li...@gmail.com>
* Fix strpos invocation with dictionary and null (#12712)
In 1b3608da7ca59d8d987804834d004e8b3e349d18 `strpos` signature was
modified to indicate it supports dictionary as input argument, but the
invoke method doesn't support them.
* docs: Update DataFusion introduction to clarify that DataFusion does provide an "out of the box" query engine (#12666)
* Update DataFusion introduction to show that DataFusion offers packaged versions for end users
* change order
* Update README.md
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* refine wording and update user guide for consistency
* prettier
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Framework for generating function docs from embedded code documentation (#12668)
* Initial work on #12432 to allow for generation of udf docs from embedded documentation in the code
* Add missing license header.
* Fixed examples.
* Fixing a really weird RustRover/wsl ... something. No clue what happened there.
* permission change
* Cargo fmt update.
* Refactored Documentation to allow it to be used in a const.
* Add documentation for syntax_example
* Refactoring Documentation based on PR feedback.
* Cargo fmt update.
* Doc update
* Fixed copy/paste error.
* Minor text updates.
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Add IMDB(JOB) Benchmark [2/N] (imdb queries) (#12529)
* imdb dataset
* cargo fmt
* Add 113 queries for IMDB(JOB)
Signed-off-by: Austin Liu <au...@gmail.com>
* Add `get_query_sql` from `query_id` string
Signed-off-by: Austin Liu <au...@gmail.com>
* Fix CSV reader & Remove Parquet partition
Signed-off-by: Austin Liu <au...@gmail.com>
* Add benchmark IMDB runner
Signed-off-by: Austin Liu <au...@gmail.com>
* Add `run_imdb` script
Signed-off-by: Austin Liu <au...@gmail.com>
* Add checker for imdb option
Signed-off-by: Austin Liu <au...@gmail.com>
* Add SLT for IMDB
Signed-off-by: Austin Liu <au...@gmail.com>
* Fix `get_query_sql()` for CI roundtrip test
Signed-off-by: Austin Liu <au...@gmail.com>
Fix `get_query_sql()` for CI roundtrip test
Signed-off-by: Austin Liu <au...@gmail.com>
Fix `get_query_sql()` for CI roundtrip test
Signed-off-by: Austin Liu <au...@gmail.com>
* Clean up
Signed-off-by: Austin Liu <au...@gmail.com>
* Add missing license
Signed-off-by: Austin Liu <au...@gmail.com>
* Add IMDB(JOB) queries `2b` to `5c`
Signed-off-by: Austin Liu <au...@gmail.com>
* Add `INCLUDE_IMDB` in CI verify-benchmark-results
Signed-off-by: Austin Liu <au...@gmail.com>
* Prepare IMDB dataset
Signed-off-by: Austin Liu <au...@gmail.com>
Prepare IMDB dataset
Signed-off-by: Austin Liu <au...@gmail.com>
* use uint as id type
* format
* Seperate `tpch` and `imdb` benchmarking CI jobs
Signed-off-by: Austin Liu <au...@gmail.com>
Fix path
Signed-off-by: Austin Liu <au...@gmail.com>
Fix path
Signed-off-by: Austin Liu <au...@gmail.com>
Remove `tpch` in `imdb` benchmark
Signed-off-by: Austin Liu <au...@gmail.com>
* Remove IMDB(JOB) slt in CI
Signed-off-by: Austin Liu <au...@gmail.com>
Remove IMDB(JOB) slt in CI
Signed-off-by: Austin Liu <au...@gmail.com>
---------
Signed-off-by: Austin Liu <au...@gmail.com>
Co-authored-by: DouPache <do...@gmail.com>
* Minor: avoid clone while calculating union equivalence properties (#12722)
* Minor: avoid clone while calculating union equivalence properties
* Update datafusion/physical-expr/src/equivalence/properties.rs
* fmt
* Simplify streaming_merge function parameters (#12719)
* simplify streaming_merge function parameters
* revert test change
* change StreamingMergeConfig into builder pattern
* Fix links on docs index page (#12750)
* Provide field and schema metadata missing on cross joins, and union with null fields. (#12729)
* test: reproducer for missing schema metadata on cross join
* fix: pass thru schema metadata on cross join
* fix: preserve metadata when transforming to view types
* test: reproducer for missing field metadata in left hand NULL field of union
* fix: preserve field metadata from right side of union
* chore: safe indexing
* Minor: Update string tests for strpos (#12739)
* Apply `type_union_resolution` to array and values (#12753)
* cleanup make array coercion rule
Signed-off-by: jayzhan211 <ja...@gmail.com>
* change to type union resolution
Signed-off-by: jayzhan211 <ja...@gmail.com>
* change value too
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix tpyo
Signed-off-by: jayzhan211 <ja...@gmail.com>
---------
Signed-off-by: jayzhan211 <ja...@gmail.com>
* Add `DocumentationBuilder::with_standard_argument` to reduce copy/paste (#12747)
* Add `DocumentationBuilder::with_standard_expression` to reduce copy/paste
* fix doc
* fix standard argument
* Update docs
* Improve documentation to explain what is different
* fix `equal_to` in `PrimitiveGroupValueBuilder` (#12758)
* fix `equal_to` in `PrimitiveGroupValueBuilder`.
* fix typo.
* add uts.
* reduce calling of `is_null`.
* Minor: doc how field name is to be set (#12757)
* Fix `equal_to` in `ByteGroupValueBuilder` (#12770)
* Fix `equal_to` in `ByteGroupValueBuilder`
* refactor null_equal_to
* Update datafusion/physical-plan/src/aggregates/group_values/group_column.rs
* Allow simplification even when nullable (#12746)
The nullable requirement seem to have been added in #1401 but as far as
I can tell they are not needed for these 2 cases.
I think this can be shown using this truth table: (generated using
datafusion-cli without this patch)
```
> CREATE TABLE t (v BOOLEAN) as values (true), (false), (NULL);
> select t.v, t2.v, t.v AND (t.v OR t2.v), t.v OR (t.v AND t2.v) from t cross join t as t2;
+-------+-------+---------------------+---------------------+
| v | v | t.v AND t.v OR t2.v | t.v OR t.v AND t2.v |
+-------+-------+---------------------+---------------------+
| true | true | true | true |
| true | false | true | true |
| true | | true | true |
| false | true | false | false |
| false | false | false | false |
| false | | false | false |
| | true | | |
| | false | | |
| | | | |
+-------+-------+---------------------+---------------------+
```
And it seems Spark applies both of these and DuckDB applies only the
first one.
* Fix unnest conjunction with selecting wildcard expression (#12760)
* fix unnest statement with wildcard expression
* add commnets
* Improve `round` scalar function unparsing for Postgres (#12744)
* Postgres: enforce required `NUMERIC` type for `round` scalar function (#34)
Includes initial support for dialects to override scalar functions unparsing
* Document scalar_function_to_sql_overrides fn
* Fix stack overflow calculating projected orderings (#12759)
* Fix stack overflow calculating projected orderings
* fix docs
* Port / Add Documentation for `VarianceSample` and `VariancePopulation` (#12742)
* Upgrade arrow/parquet to `53.1.0` / fix clippy (#12724)
* Update to arrow/parquet 53.1.0
* Update some API
* update for changed file sizes
* Use non deprecated APIs
* Use ParquetMetadataReader from @etseidl
* remove upstreamed implementation
* Update CSV schema
* Use upstream is_null and is_not_null kernels
* feat: add support for Substrait ExtendedExpression (#12728)
* Add support for serializing and deserializing Substrait ExtendedExpr message
* Address clippy reviews
* Reuse existing rename method
* Transformed::new_transformed: Fix documentation formatting (#12787)
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* fix: Correct results for grouping sets when columns contain nulls (#12571)
* Fix grouping sets behavior when data contains nulls
* PR suggestion comment
* Update new test case
* Add grouping_id to the logical plan
* Add doc comment next to INTERNAL_GROUPING_ID
* Fix unparsing of Aggregate with grouping sets
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Migrate documentation for all string functions from scalar_functions.md to code (#12775)
* Added documentation for string and unicode functions.
* Fixed issues with aliases.
* Cargo fmt.
* Minor doc fixes.
* Update docs for var_pop/samp
---------
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
* Account for constant equivalence properties in union, tests (#12562)
* Minor: clarify comment about empty dependencies (#12786)
* Introduce Signature::String and return error if input of `strpos` is integer (#12751)
* fix sig
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix error
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix all signature
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix all signature
Signed-off-by: jayzhan211 <ja...@gmail.com>
* change default type
Signed-off-by: jayzhan211 <ja...@gmail.com>
* clippy
Signed-off-by: jayzhan211 <ja...@gmail.com>
* fix docs
Signed-off-by: jayzhan211 <ja...@gmail.com>
* rm deadcode
Signed-off-by: jayzhan211 <ja...@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <ja...@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <ja...@gmail.com>
* rm test
Signed-off-by: jayzhan211 <ja...@gmail.com>
---------
Signed-off-by: jayzhan211 <ja...@gmail.com>
* Minor: improve docs on MovingMin/MovingMax (#12790)
* Add slt tests (#12721)
---------
Signed-off-by: jayzhan211 <ja...@gmail.com>
Signed-off-by: Austin Liu <au...@gmail.com>
Co-authored-by: OussamaSaoudi <45...@users.noreply.github.com>
Co-authored-by: Jonah Gao <jo...@msn.com>
Co-authored-by: Dmitrii Blaginin <dm...@blaginin.me>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
Co-authored-by: Tomoaki Kawada <ka...@kmckk.co.jp>
Co-authored-by: Piotr Findeisen <pi...@gmail.com>
Co-authored-by: Jay Zhan <ja...@gmail.com>
Co-authored-by: HuSen <hu...@gmail.com>
Co-authored-by: Emil Ejbyfeldt <em...@gmail.com>
Co-authored-by: Simon Vandel Sillesen <si...@gmail.com>
Co-authored-by: Jax Liu <li...@gmail.com>
Co-authored-by: Austin Liu <au...@gmail.com>
Co-authored-by: JonasDev1 <js...@googlemail.com>
Co-authored-by: jcsherin <ja...@protoship.io>
Co-authored-by: Sergei Grebnov <se...@gmail.com>
Co-authored-by: Andy Grove <ag...@apache.org>
Co-authored-by: Bruce Ritchie <br...@veeva.com>
Co-authored-by: DouPache <do...@gmail.com>
Co-authored-by: mertak-synnada <me...@gmail.com>
Co-authored-by: Bryce Mecum <pe...@gmail.com>
Co-authored-by: wiedld <wi...@users.noreply.github.com>
Co-authored-by: kamille <ca...@antgroup.com>
Co-authored-by: Weston Pace <we...@gmail.com>
Co-authored-by: Val Lorentz <vl...@softwareheritage.org>
---
.github/workflows/rust.yml | 7 +-
Cargo.toml | 18 +-
README.md | 17 +-
benchmarks/bench.sh | 14 +
benchmarks/queries/imdb/10a.sql | 1 +
benchmarks/queries/imdb/10b.sql | 1 +
benchmarks/queries/imdb/10c.sql | 1 +
benchmarks/queries/imdb/11a.sql | 1 +
benchmarks/queries/imdb/11b.sql | 1 +
benchmarks/queries/imdb/11c.sql | 1 +
benchmarks/queries/imdb/11d.sql | 1 +
benchmarks/queries/imdb/12a.sql | 1 +
benchmarks/queries/imdb/12b.sql | 1 +
benchmarks/queries/imdb/12c.sql | 1 +
benchmarks/queries/imdb/13a.sql | 1 +
benchmarks/queries/imdb/13b.sql | 1 +
benchmarks/queries/imdb/13c.sql | 1 +
benchmarks/queries/imdb/13d.sql | 1 +
benchmarks/queries/imdb/14a.sql | 1 +
benchmarks/queries/imdb/14b.sql | 1 +
benchmarks/queries/imdb/14c.sql | 1 +
benchmarks/queries/imdb/15a.sql | 1 +
benchmarks/queries/imdb/15b.sql | 1 +
benchmarks/queries/imdb/15c.sql | 1 +
benchmarks/queries/imdb/15d.sql | 1 +
benchmarks/queries/imdb/16a.sql | 1 +
benchmarks/queries/imdb/16b.sql | 1 +
benchmarks/queries/imdb/16c.sql | 1 +
benchmarks/queries/imdb/16d.sql | 1 +
benchmarks/queries/imdb/17a.sql | 1 +
benchmarks/queries/imdb/17b.sql | 1 +
benchmarks/queries/imdb/17c.sql | 1 +
benchmarks/queries/imdb/17d.sql | 1 +
benchmarks/queries/imdb/17e.sql | 1 +
benchmarks/queries/imdb/17f.sql | 1 +
benchmarks/queries/imdb/18a.sql | 1 +
benchmarks/queries/imdb/18b.sql | 1 +
benchmarks/queries/imdb/18c.sql | 1 +
benchmarks/queries/imdb/19a.sql | 1 +
benchmarks/queries/imdb/19b.sql | 1 +
benchmarks/queries/imdb/19c.sql | 1 +
benchmarks/queries/imdb/19d.sql | 1 +
benchmarks/queries/imdb/1a.sql | 1 +
benchmarks/queries/imdb/1b.sql | 1 +
benchmarks/queries/imdb/1c.sql | 1 +
benchmarks/queries/imdb/1d.sql | 1 +
benchmarks/queries/imdb/20a.sql | 1 +
benchmarks/queries/imdb/20b.sql | 1 +
benchmarks/queries/imdb/20c.sql | 1 +
benchmarks/queries/imdb/21a.sql | 1 +
benchmarks/queries/imdb/21b.sql | 1 +
benchmarks/queries/imdb/21c.sql | 1 +
benchmarks/queries/imdb/22a.sql | 1 +
benchmarks/queries/imdb/22b.sql | 1 +
benchmarks/queries/imdb/22c.sql | 1 +
benchmarks/queries/imdb/22d.sql | 1 +
benchmarks/queries/imdb/23a.sql | 1 +
benchmarks/queries/imdb/23b.sql | 1 +
benchmarks/queries/imdb/23c.sql | 1 +
benchmarks/queries/imdb/24a.sql | 1 +
benchmarks/queries/imdb/24b.sql | 1 +
benchmarks/queries/imdb/25a.sql | 1 +
benchmarks/queries/imdb/25b.sql | 1 +
benchmarks/queries/imdb/25c.sql | 1 +
benchmarks/queries/imdb/26a.sql | 1 +
benchmarks/queries/imdb/26b.sql | 1 +
benchmarks/queries/imdb/26c.sql | 1 +
benchmarks/queries/imdb/27a.sql | 1 +
benchmarks/queries/imdb/27b.sql | 1 +
benchmarks/queries/imdb/27c.sql | 1 +
benchmarks/queries/imdb/28a.sql | 1 +
benchmarks/queries/imdb/28b.sql | 1 +
benchmarks/queries/imdb/28c.sql | 1 +
benchmarks/queries/imdb/29a.sql | 1 +
benchmarks/queries/imdb/29b.sql | 1 +
benchmarks/queries/imdb/29c.sql | 1 +
benchmarks/queries/imdb/2a.sql | 1 +
benchmarks/queries/imdb/2b.sql | 1 +
benchmarks/queries/imdb/2c.sql | 1 +
benchmarks/queries/imdb/2d.sql | 1 +
benchmarks/queries/imdb/30a.sql | 1 +
benchmarks/queries/imdb/30b.sql | 1 +
benchmarks/queries/imdb/30c.sql | 1 +
benchmarks/queries/imdb/31a.sql | 1 +
benchmarks/queries/imdb/31b.sql | 1 +
benchmarks/queries/imdb/31c.sql | 1 +
benchmarks/queries/imdb/32a.sql | 1 +
benchmarks/queries/imdb/32b.sql | 1 +
benchmarks/queries/imdb/33a.sql | 1 +
benchmarks/queries/imdb/33b.sql | 1 +
benchmarks/queries/imdb/33c.sql | 1 +
benchmarks/queries/imdb/3a.sql | 1 +
benchmarks/queries/imdb/3b.sql | 1 +
benchmarks/queries/imdb/3c.sql | 1 +
benchmarks/queries/imdb/4a.sql | 1 +
benchmarks/queries/imdb/4b.sql | 1 +
benchmarks/queries/imdb/4c.sql | 1 +
benchmarks/queries/imdb/5a.sql | 1 +
benchmarks/queries/imdb/5b.sql | 1 +
benchmarks/queries/imdb/5c.sql | 1 +
benchmarks/queries/imdb/6a.sql | 1 +
benchmarks/queries/imdb/6b.sql | 1 +
benchmarks/queries/imdb/6c.sql | 1 +
benchmarks/queries/imdb/6d.sql | 1 +
benchmarks/queries/imdb/6e.sql | 1 +
benchmarks/queries/imdb/6f.sql | 1 +
benchmarks/queries/imdb/7a.sql | 1 +
benchmarks/queries/imdb/7b.sql | 1 +
benchmarks/queries/imdb/7c.sql | 1 +
benchmarks/queries/imdb/8a.sql | 1 +
benchmarks/queries/imdb/8b.sql | 1 +
benchmarks/queries/imdb/8c.sql | 1 +
benchmarks/queries/imdb/8d.sql | 1 +
benchmarks/queries/imdb/9a.sql | 1 +
benchmarks/queries/imdb/9b.sql | 1 +
benchmarks/queries/imdb/9c.sql | 1 +
benchmarks/queries/imdb/9d.sql | 1 +
benchmarks/src/bin/dfbench.rs | 4 +-
benchmarks/src/bin/imdb.rs | 11 +
benchmarks/src/imdb/convert.rs | 6 +-
benchmarks/src/imdb/mod.rs | 75 +-
benchmarks/src/imdb/run.rs | 827 ++++++++++++
datafusion-cli/Cargo.lock | 419 +++---
datafusion/common/src/dfschema.rs | 12 +-
datafusion/common/src/tree_node.rs | 2 +-
datafusion/core/src/bin/print_functions_docs.rs | 266 ++++
datafusion/core/src/catalog_common/mod.rs | 4 +-
datafusion/core/src/dataframe/mod.rs | 63 +-
.../datasource/avro_to_arrow/arrow_array_reader.rs | 89 +-
datafusion/core/src/datasource/dynamic_file.rs | 13 +-
datafusion/core/src/datasource/file_format/csv.rs | 4 +-
datafusion/core/src/datasource/file_format/mod.rs | 18 +-
.../core/src/datasource/file_format/parquet.rs | 97 +-
datafusion/core/src/datasource/listing/table.rs | 36 +-
datafusion/core/src/physical_planner.rs | 18 +-
datafusion/core/tests/expr_api/mod.rs | 8 +-
.../fuzz_cases/sort_preserving_repartition_fuzz.rs | 19 +-
.../core/tests/user_defined/user_defined_plan.rs | 4 +
datafusion/expr-common/src/signature.rs | 18 +-
datafusion/expr-common/src/type_coercion/binary.rs | 24 +-
datafusion/expr/Cargo.toml | 1 +
datafusion/expr/src/lib.rs | 10 +-
datafusion/expr/src/logical_plan/builder.rs | 5 +-
datafusion/expr/src/logical_plan/extension.rs | 24 +
datafusion/expr/src/logical_plan/plan.rs | 59 +-
datafusion/expr/src/type_coercion/functions.rs | 228 ++--
datafusion/expr/src/udaf.rs | 81 +-
datafusion/expr/src/udf.rs | 142 +-
datafusion/expr/src/udf_docs.rs | 211 +++
datafusion/expr/src/udwf.rs | 90 +-
datafusion/expr/src/utils.rs | 22 +-
datafusion/functions-aggregate/Cargo.toml | 2 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 87 +-
datafusion/functions-aggregate/src/macros.rs | 17 -
datafusion/functions-aggregate/src/min_max.rs | 43 +-
datafusion/functions-aggregate/src/variance.rs | 47 +-
datafusion/functions-nested/src/make_array.rs | 54 +-
datafusion/functions-window/Cargo.toml | 1 +
datafusion/functions-window/src/lib.rs | 2 +
datafusion/functions-window/src/macros.rs | 674 ++++++++++
datafusion/functions-window/src/row_number.rs | 58 +-
datafusion/functions/Cargo.toml | 5 +
datafusion/functions/benches/encoding.rs | 53 +
datafusion/functions/src/core/coalesce.rs | 28 +-
datafusion/functions/src/crypto/sha224.rs | 27 +-
datafusion/functions/src/datetime/to_date.rs | 60 +-
datafusion/functions/src/encoding/inner.rs | 139 +-
datafusion/functions/src/lib.rs | 3 -
datafusion/functions/src/macros.rs | 6 +-
datafusion/functions/src/math/log.rs | 27 +-
datafusion/functions/src/math/nans.rs | 8 +-
datafusion/functions/src/math/power.rs | 8 +-
datafusion/functions/src/regex/regexplike.rs | 64 +-
datafusion/functions/src/regex/regexpmatch.rs | 13 +-
datafusion/functions/src/regex/regexpreplace.rs | 10 +-
datafusion/functions/src/regexp_common.rs | 123 --
datafusion/functions/src/string/ascii.rs | 49 +-
datafusion/functions/src/string/bit_length.rs | 47 +-
datafusion/functions/src/string/btrim.rs | 56 +-
datafusion/functions/src/string/chr.rs | 39 +-
datafusion/functions/src/string/concat.rs | 40 +-
datafusion/functions/src/string/concat_ws.rs | 44 +-
datafusion/functions/src/string/contains.rs | 218 +---
datafusion/functions/src/string/ends_with.rs | 56 +-
datafusion/functions/src/string/initcap.rs | 43 +-
datafusion/functions/src/string/levenshtein.rs | 42 +-
datafusion/functions/src/string/lower.rs | 48 +-
datafusion/functions/src/string/ltrim.rs | 59 +-
datafusion/functions/src/string/octet_length.rs | 47 +-
datafusion/functions/src/string/overlay.rs | 50 +-
datafusion/functions/src/string/repeat.rs | 49 +-
datafusion/functions/src/string/replace.rs | 47 +-
datafusion/functions/src/string/rtrim.rs | 59 +-
datafusion/functions/src/string/split_part.rs | 55 +-
datafusion/functions/src/string/starts_with.rs | 50 +-
datafusion/functions/src/string/to_hex.rs | 36 +-
datafusion/functions/src/string/upper.rs | 41 +-
datafusion/functions/src/string/uuid.rs | 30 +-
.../functions/src/unicode/character_length.rs | 37 +-
datafusion/functions/src/unicode/find_in_set.rs | 37 +-
datafusion/functions/src/unicode/left.rs | 38 +-
datafusion/functions/src/unicode/lpad.rs | 41 +-
datafusion/functions/src/unicode/reverse.rs | 38 +-
datafusion/functions/src/unicode/right.rs | 35 +-
datafusion/functions/src/unicode/rpad.rs | 40 +-
datafusion/functions/src/unicode/strpos.rs | 58 +-
datafusion/functions/src/unicode/substr.rs | 35 +-
datafusion/functions/src/unicode/substrindex.rs | 46 +-
datafusion/functions/src/unicode/translate.rs | 35 +-
.../optimizer/src/analyzer/count_wildcard_rule.rs | 8 +-
datafusion/optimizer/src/analyzer/subquery.rs | 4 +
datafusion/optimizer/src/analyzer/type_coercion.rs | 25 -
.../optimizer/src/optimize_projections/mod.rs | 8 +
datafusion/optimizer/src/push_down_filter.rs | 4 +
datafusion/optimizer/src/push_down_limit.rs | 249 +++-
.../src/simplify_expressions/expr_simplifier.rs | 40 +-
.../optimizer/src/single_distinct_to_groupby.rs | 6 +-
datafusion/optimizer/src/test/user_defined.rs | 4 +
.../optimizer/tests/optimizer_integration.rs | 4 +-
datafusion/physical-expr-common/src/sort_expr.rs | 7 +
datafusion/physical-expr/src/equivalence/class.rs | 50 +-
.../physical-expr/src/equivalence/ordering.rs | 18 +-
.../physical-expr/src/equivalence/properties.rs | 1364 +++++++++++++-------
datafusion/physical-expr/src/expressions/binary.rs | 8 +-
.../physical-expr/src/expressions/is_not_null.rs | 2 +-
.../physical-expr/src/expressions/is_null.rs | 77 +-
.../src/aggregates/group_values/column.rs | 10 +-
.../src/aggregates/group_values/group_column.rs | 360 ++++--
datafusion/physical-plan/src/aggregates/mod.rs | 370 +++---
.../physical-plan/src/aggregates/row_hash.rs | 36 +-
datafusion/physical-plan/src/joins/cross_join.rs | 13 +-
datafusion/physical-plan/src/repartition/mod.rs | 20 +-
datafusion/physical-plan/src/sorts/mod.rs | 1 -
datafusion/physical-plan/src/sorts/sort.rs | 38 +-
.../src/sorts/sort_preserving_merge.rs | 39 +-
.../physical-plan/src/sorts/streaming_merge.rs | 151 ++-
datafusion/physical-plan/src/union.rs | 11 +-
.../proto/tests/cases/roundtrip_logical_plan.rs | 4 +
datafusion/sql/src/expr/function.rs | 12 +
datafusion/sql/src/parser.rs | 39 +-
datafusion/sql/src/planner.rs | 6 +-
datafusion/sql/src/statement.rs | 3 +-
datafusion/sql/src/unparser/dialect.rs | 119 +-
datafusion/sql/src/unparser/expr.rs | 198 ++-
datafusion/sql/src/unparser/plan.rs | 25 +-
datafusion/sql/src/unparser/utils.rs | 223 +++-
datafusion/sql/src/utils.rs | 4 +-
datafusion/sql/tests/cases/plan_to_sql.rs | 21 +
datafusion/sql/tests/sql_integration.rs | 7 +
datafusion/sqllogictest/src/test_context.rs | 8 +-
datafusion/sqllogictest/test_files/aggregate.slt | 40 +-
datafusion/sqllogictest/test_files/array.slt | 2 +-
.../test_files/create_external_table.slt | 12 +
datafusion/sqllogictest/test_files/cse.slt | 16 +-
.../sqllogictest/test_files/dynamic_file.slt | 167 ++-
datafusion/sqllogictest/test_files/errors.slt | 2 +-
datafusion/sqllogictest/test_files/functions.slt | 10 +
datafusion/sqllogictest/test_files/group_by.slt | 11 +-
datafusion/sqllogictest/test_files/join.slt | 6 +-
.../join_disable_repartition_joins.slt.temp | 26 -
datafusion/sqllogictest/test_files/map.slt | 32 +-
datafusion/sqllogictest/test_files/metadata.slt | 31 +-
datafusion/sqllogictest/test_files/order.slt | 63 +
.../sqllogictest/test_files/repartition_scan.slt | 8 +-
datafusion/sqllogictest/test_files/scalar.slt | 4 +-
datafusion/sqllogictest/test_files/select.slt | 10 +-
.../test_files/string/dictionary_utf8.slt | 30 -
.../test_files/string/init_data.slt.part | 1 -
.../test_files/string/large_string.slt | 17 -
.../sqllogictest/test_files/string/string.slt | 47 -
.../test_files/string/string_query.slt.part | 94 +-
.../sqllogictest/test_files/string/string_view.slt | 71 +-
datafusion/sqllogictest/test_files/unnest.slt | 54 +-
datafusion/substrait/src/logical_plan/consumer.rs | 203 ++-
datafusion/substrait/src/logical_plan/producer.rs | 230 +++-
.../tests/cases/roundtrip_logical_plan.rs | 9 +-
dev/update_config_docs.sh | 6 +-
dev/update_function_docs.sh | 284 ++++
docs/source/index.rst | 15 +-
docs/source/user-guide/expressions.md | 4 +-
docs/source/user-guide/sql/aggregate_functions.md | 42 -
.../user-guide/sql/aggregate_functions_new.md | 124 ++
docs/source/user-guide/sql/data_types.md | 2 +-
docs/source/user-guide/sql/index.rst | 3 +
docs/source/user-guide/sql/scalar_functions.md | 626 +--------
docs/source/user-guide/sql/scalar_functions_new.md | 1172 +++++++++++++++++
docs/source/user-guide/sql/window_functions_new.md | 161 +++
287 files changed, 9951 insertions(+), 3563 deletions(-)
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index edaa49ec6e..4527d047e4 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -521,7 +521,7 @@ jobs:
run: taplo format --check
config-docs-check:
- name: check configs.md is up-to-date
+ name: check configs.md and ***_functions.md is up-to-date
needs: [ linux-build-lib ]
runs-on: ubuntu-latest
container:
@@ -542,6 +542,11 @@ jobs:
# If you encounter an error, run './dev/update_config_docs.sh' and commit
./dev/update_config_docs.sh
git diff --exit-code
+ - name: Check if any of the ***_functions.md has been modified
+ run: |
+ # If you encounter an error, run './dev/update_function_docs.sh' and commit
+ ./dev/update_function_docs.sh
+ git diff --exit-code
# Verify MSRV for the crates which are directly used by other projects:
# - datafusion
diff --git a/Cargo.toml b/Cargo.toml
index b8bf83a5ab..448607257c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -70,22 +70,22 @@ version = "42.0.0"
ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
-arrow = { version = "53.0.0", features = [
+arrow = { version = "53.1.0", features = [
"prettyprint",
] }
-arrow-array = { version = "53.0.0", default-features = false, features = [
+arrow-array = { version = "53.1.0", default-features = false, features = [
"chrono-tz",
] }
-arrow-buffer = { version = "53.0.0", default-features = false }
-arrow-flight = { version = "53.0.0", features = [
+arrow-buffer = { version = "53.1.0", default-features = false }
+arrow-flight = { version = "53.1.0", features = [
"flight-sql-experimental",
] }
-arrow-ipc = { version = "53.0.0", default-features = false, features = [
+arrow-ipc = { version = "53.1.0", default-features = false, features = [
"lz4",
] }
-arrow-ord = { version = "53.0.0", default-features = false }
-arrow-schema = { version = "53.0.0", default-features = false }
-arrow-string = { version = "53.0.0", default-features = false }
+arrow-ord = { version = "53.1.0", default-features = false }
+arrow-schema = { version = "53.1.0", default-features = false }
+arrow-string = { version = "53.1.0", default-features = false }
async-trait = "0.1.73"
bigdecimal = "=0.4.1"
bytes = "1.4"
@@ -126,7 +126,7 @@ log = "^0.4"
num_cpus = "1.13.0"
object_store = { version = "0.11.0", default-features = false }
parking_lot = "0.12"
-parquet = { version = "53.0.0", default-features = false, features = [
+parquet = { version = "53.1.0", default-features = false, features = [
"arrow",
"async",
"object_store",
diff --git a/README.md b/README.md
index bb8526c24e..5d0b096c1d 100644
--- a/README.md
+++ b/README.md
@@ -42,14 +42,25 @@
</a>
DataFusion is an extensible query engine written in [Rust] that
-uses [Apache Arrow] as its in-memory format. DataFusion's target users are
+uses [Apache Arrow] as its in-memory format.
+
+The DataFusion libraries in this repository are used to build data-centric system software. DataFusion also provides the
+following subprojects, which are packaged versions of DataFusion intended for end users.
+
+- [DataFusion Python](https://github.com/apache/datafusion-python/) offers a Python interface for SQL and DataFrame
+ queries.
+- [DataFusion Ray](https://github.com/apache/datafusion-ray/) provides a distributed version of DataFusion that scales
+ out on Ray clusters.
+- [DataFusion Comet](https://github.com/apache/datafusion-comet/) is an accelerator for Apache Spark based on
+ DataFusion.
+
+The target audience for the DataFusion crates in this repository are
developers building fast and feature rich database and analytic systems,
customized to particular workloads. See [use cases] for examples.
-"Out of the box," DataFusion offers [SQL] and [`Dataframe`] APIs,
+DataFusion offers [SQL] and [`Dataframe`] APIs,
excellent [performance], built-in support for CSV, Parquet, JSON, and Avro,
extensive customization, and a great community.
-[Python Bindings] are also available.
DataFusion features a full query planner, a columnar, streaming, multi-threaded,
vectorized execution engine, and partitioned data sources. You can
diff --git a/benchmarks/bench.sh b/benchmarks/bench.sh
index 24efab6c6c..70faa9ef2b 100755
--- a/benchmarks/bench.sh
+++ b/benchmarks/bench.sh
@@ -211,6 +211,7 @@ main() {
run_clickbench_1
run_clickbench_partitioned
run_clickbench_extended
+ run_imdb
;;
tpch)
run_tpch "1"
@@ -239,6 +240,9 @@ main() {
clickbench_extended)
run_clickbench_extended
;;
+ imdb)
+ run_imdb
+ ;;
*)
echo "Error: unknown benchmark '$BENCHMARK' for run"
usage
@@ -510,6 +514,16 @@ data_imdb() {
fi
}
+# Runs the imdb benchmark
+run_imdb() {
+ IMDB_DIR="${DATA_DIR}/imdb"
+
+ RESULTS_FILE="${RESULTS_DIR}/imdb.json"
+ echo "RESULTS_FILE: ${RESULTS_FILE}"
+ echo "Running imdb benchmark..."
+ $CARGO_COMMAND --bin imdb -- benchmark datafusion --iterations 5 --path "${IMDB_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" --format parquet -o "${RESULTS_FILE}"
+}
+
diff --git a/benchmarks/queries/imdb/10a.sql b/benchmarks/queries/imdb/10a.sql
new file mode 100644
index 0000000000..95b049b774
--- /dev/null
+++ b/benchmarks/queries/imdb/10a.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS uncredited_voiced_character, MIN(t.title) AS russian_movie FROM char_name AS chn, cast_info AS ci, company_name AS cn, company_type AS ct, movie_companies AS mc, role_type AS rt, title AS t WHERE ci.note like '%(voice)%' and ci.note like '%(uncredited)%' AND cn.country_code = '[ru]' AND rt.role = 'actor' AND t.production_year > 2005 AND t.id = mc.movie_id AND t.id = ci.movie_id AND ci.movie_id = mc.movie_id AND chn.id = ci.person_role_id AND rt.id = ci.role_id [...]
diff --git a/benchmarks/queries/imdb/10b.sql b/benchmarks/queries/imdb/10b.sql
new file mode 100644
index 0000000000..c321536314
--- /dev/null
+++ b/benchmarks/queries/imdb/10b.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS character, MIN(t.title) AS russian_mov_with_actor_producer FROM char_name AS chn, cast_info AS ci, company_name AS cn, company_type AS ct, movie_companies AS mc, role_type AS rt, title AS t WHERE ci.note like '%(producer)%' AND cn.country_code = '[ru]' AND rt.role = 'actor' AND t.production_year > 2010 AND t.id = mc.movie_id AND t.id = ci.movie_id AND ci.movie_id = mc.movie_id AND chn.id = ci.person_role_id AND rt.id = ci.role_id AND cn.id = mc.company_id AND c [...]
diff --git a/benchmarks/queries/imdb/10c.sql b/benchmarks/queries/imdb/10c.sql
new file mode 100644
index 0000000000..b862cf4fa7
--- /dev/null
+++ b/benchmarks/queries/imdb/10c.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS character, MIN(t.title) AS movie_with_american_producer FROM char_name AS chn, cast_info AS ci, company_name AS cn, company_type AS ct, movie_companies AS mc, role_type AS rt, title AS t WHERE ci.note like '%(producer)%' AND cn.country_code = '[us]' AND t.production_year > 1990 AND t.id = mc.movie_id AND t.id = ci.movie_id AND ci.movie_id = mc.movie_id AND chn.id = ci.person_role_id AND rt.id = ci.role_id AND cn.id = mc.company_id AND ct.id = mc.company_type_id;
diff --git a/benchmarks/queries/imdb/11a.sql b/benchmarks/queries/imdb/11a.sql
new file mode 100644
index 0000000000..f835968e90
--- /dev/null
+++ b/benchmarks/queries/imdb/11a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS from_company, MIN(lt.link) AS movie_link_type, MIN(t.title) AS non_polish_sequel_movie FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND ct.kind ='production companies' AND k.keyword ='sequel' AND lt.link LIKE '%follow%' AND mc.note IS NULL AND t.production_year BETWEEN 1950 AND [...]
diff --git a/benchmarks/queries/imdb/11b.sql b/benchmarks/queries/imdb/11b.sql
new file mode 100644
index 0000000000..2411e19ea6
--- /dev/null
+++ b/benchmarks/queries/imdb/11b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS from_company, MIN(lt.link) AS movie_link_type, MIN(t.title) AS sequel_movie FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND ct.kind ='production companies' AND k.keyword ='sequel' AND lt.link LIKE '%follows%' AND mc.note IS NULL AND t.production_year = 1998 and t.title like '% [...]
diff --git a/benchmarks/queries/imdb/11c.sql b/benchmarks/queries/imdb/11c.sql
new file mode 100644
index 0000000000..3bf7946789
--- /dev/null
+++ b/benchmarks/queries/imdb/11c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS from_company, MIN(mc.note) AS production_note, MIN(t.title) AS movie_based_on_book FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' and (cn.name like '20th Century Fox%' or cn.name like 'Twentieth Century Fox%') AND ct.kind != 'production companies' and ct.kind is not NULL AND k.keyword in ('sequel', 'revenge', 'based-on-novel') AND [...]
diff --git a/benchmarks/queries/imdb/11d.sql b/benchmarks/queries/imdb/11d.sql
new file mode 100644
index 0000000000..0bc33e1d6e
--- /dev/null
+++ b/benchmarks/queries/imdb/11d.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS from_company, MIN(mc.note) AS production_note, MIN(t.title) AS movie_based_on_book FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND ct.kind != 'production companies' and ct.kind is not NULL AND k.keyword in ('sequel', 'revenge', 'based-on-novel') AND mc.note is not NULL AND t.production_year > 1950 AND lt.id = ml.link_type_id A [...]
diff --git a/benchmarks/queries/imdb/12a.sql b/benchmarks/queries/imdb/12a.sql
new file mode 100644
index 0000000000..22add74bd5
--- /dev/null
+++ b/benchmarks/queries/imdb/12a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS drama_horror_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, title AS t WHERE cn.country_code = '[us]' AND ct.kind = 'production companies' AND it1.info = 'genres' AND it2.info = 'rating' AND mi.info in ('Drama', 'Horror') AND mi_idx.info > '8.0' AND t.production_year between 2005 and 2008 AND t.id [...]
diff --git a/benchmarks/queries/imdb/12b.sql b/benchmarks/queries/imdb/12b.sql
new file mode 100644
index 0000000000..fc30ad550d
--- /dev/null
+++ b/benchmarks/queries/imdb/12b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS budget, MIN(t.title) AS unsuccsessful_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, title AS t WHERE cn.country_code ='[us]' AND ct.kind is not NULL and (ct.kind ='production companies' or ct.kind = 'distributors') AND it1.info ='budget' AND it2.info ='bottom 10 rank' AND t.production_year >2000 AND (t.title LIKE 'Birdemic%' OR t.title LIKE '%Movie%') AND t. [...]
diff --git a/benchmarks/queries/imdb/12c.sql b/benchmarks/queries/imdb/12c.sql
new file mode 100644
index 0000000000..64a340b238
--- /dev/null
+++ b/benchmarks/queries/imdb/12c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS mainstream_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, title AS t WHERE cn.country_code = '[us]' AND ct.kind = 'production companies' AND it1.info = 'genres' AND it2.info = 'rating' AND mi.info in ('Drama', 'Horror', 'Western', 'Family') AND mi_idx.info > '7.0' AND t.production_year between 200 [...]
diff --git a/benchmarks/queries/imdb/13a.sql b/benchmarks/queries/imdb/13a.sql
new file mode 100644
index 0000000000..95eb439d1e
--- /dev/null
+++ b/benchmarks/queries/imdb/13a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS release_date, MIN(miidx.info) AS rating, MIN(t.title) AS german_movie FROM company_name AS cn, company_type AS ct, info_type AS it, info_type AS it2, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS miidx, title AS t WHERE cn.country_code ='[de]' AND ct.kind ='production companies' AND it.info ='rating' AND it2.info ='release dates' AND kt.kind ='movie' AND mi.movie_id = t.id AND it2.id = mi.info_type_id AND kt.id = t.kind_id AND mc.movie [...]
diff --git a/benchmarks/queries/imdb/13b.sql b/benchmarks/queries/imdb/13b.sql
new file mode 100644
index 0000000000..4b6f75ab0a
--- /dev/null
+++ b/benchmarks/queries/imdb/13b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(miidx.info) AS rating, MIN(t.title) AS movie_about_winning FROM company_name AS cn, company_type AS ct, info_type AS it, info_type AS it2, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS miidx, title AS t WHERE cn.country_code ='[us]' AND ct.kind ='production companies' AND it.info ='rating' AND it2.info ='release dates' AND kt.kind ='movie' AND t.title != '' AND (t.title LIKE '%Champion%' OR t.title LIKE '%Loser% [...]
diff --git a/benchmarks/queries/imdb/13c.sql b/benchmarks/queries/imdb/13c.sql
new file mode 100644
index 0000000000..9e8c92327b
--- /dev/null
+++ b/benchmarks/queries/imdb/13c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(miidx.info) AS rating, MIN(t.title) AS movie_about_winning FROM company_name AS cn, company_type AS ct, info_type AS it, info_type AS it2, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS miidx, title AS t WHERE cn.country_code ='[us]' AND ct.kind ='production companies' AND it.info ='rating' AND it2.info ='release dates' AND kt.kind ='movie' AND t.title != '' AND (t.title LIKE 'Champion%' OR t.title LIKE 'Loser%') [...]
diff --git a/benchmarks/queries/imdb/13d.sql b/benchmarks/queries/imdb/13d.sql
new file mode 100644
index 0000000000..a8bc567cab
--- /dev/null
+++ b/benchmarks/queries/imdb/13d.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(miidx.info) AS rating, MIN(t.title) AS movie FROM company_name AS cn, company_type AS ct, info_type AS it, info_type AS it2, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS miidx, title AS t WHERE cn.country_code ='[us]' AND ct.kind ='production companies' AND it.info ='rating' AND it2.info ='release dates' AND kt.kind ='movie' AND mi.movie_id = t.id AND it2.id = mi.info_type_id AND kt.id = t.kind_id AND mc.movie_i [...]
diff --git a/benchmarks/queries/imdb/14a.sql b/benchmarks/queries/imdb/14a.sql
new file mode 100644
index 0000000000..af1a7c8983
--- /dev/null
+++ b/benchmarks/queries/imdb/14a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS northern_dark_movie FROM info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind = 'movie' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German', 'USA', 'American') AND m [...]
diff --git a/benchmarks/queries/imdb/14b.sql b/benchmarks/queries/imdb/14b.sql
new file mode 100644
index 0000000000..c606ebc73d
--- /dev/null
+++ b/benchmarks/queries/imdb/14b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS western_dark_production FROM info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title') AND kt.kind = 'movie' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German', 'USA', 'American') AND mi_idx.info > '6. [...]
diff --git a/benchmarks/queries/imdb/14c.sql b/benchmarks/queries/imdb/14c.sql
new file mode 100644
index 0000000000..2a6dffde26
--- /dev/null
+++ b/benchmarks/queries/imdb/14c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS north_european_dark_production FROM info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it1.info = 'countries' AND it2.info = 'rating' AND k.keyword is not null and k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind in ('movie', 'episode') AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Danis [...]
diff --git a/benchmarks/queries/imdb/15a.sql b/benchmarks/queries/imdb/15a.sql
new file mode 100644
index 0000000000..1d052f0044
--- /dev/null
+++ b/benchmarks/queries/imdb/15a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS release_date, MIN(t.title) AS internet_movie FROM aka_title AS at, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cn.country_code = '[us]' AND it1.info = 'release dates' AND mc.note like '%(200%)%' and mc.note like '%(worldwide)%' AND mi.note like '%internet%' AND mi.info like 'USA:% 200%' AND t.production_year > 2000 AND t.id = at.movie_id AND t.id = mi.mo [...]
diff --git a/benchmarks/queries/imdb/15b.sql b/benchmarks/queries/imdb/15b.sql
new file mode 100644
index 0000000000..21c81358fa
--- /dev/null
+++ b/benchmarks/queries/imdb/15b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS release_date, MIN(t.title) AS youtube_movie FROM aka_title AS at, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cn.country_code = '[us]' and cn.name = 'YouTube' AND it1.info = 'release dates' AND mc.note like '%(200%)%' and mc.note like '%(worldwide)%' AND mi.note like '%internet%' AND mi.info like 'USA:% 200%' AND t.production_year between 2005 and 2010 A [...]
diff --git a/benchmarks/queries/imdb/15c.sql b/benchmarks/queries/imdb/15c.sql
new file mode 100644
index 0000000000..2d08c52039
--- /dev/null
+++ b/benchmarks/queries/imdb/15c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS release_date, MIN(t.title) AS modern_american_internet_movie FROM aka_title AS at, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cn.country_code = '[us]' AND it1.info = 'release dates' AND mi.note like '%internet%' AND mi.info is not NULL and (mi.info like 'USA:% 199%' or mi.info like 'USA:% 200%') AND t.production_year > 1990 AND t.id = at.movie_id AND t.i [...]
diff --git a/benchmarks/queries/imdb/15d.sql b/benchmarks/queries/imdb/15d.sql
new file mode 100644
index 0000000000..040e9815d8
--- /dev/null
+++ b/benchmarks/queries/imdb/15d.sql
@@ -0,0 +1 @@
+SELECT MIN(at.title) AS aka_title, MIN(t.title) AS internet_movie_title FROM aka_title AS at, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cn.country_code = '[us]' AND it1.info = 'release dates' AND mi.note like '%internet%' AND t.production_year > 1990 AND t.id = at.movie_id AND t.id = mi.movie_id AND t.id = mk.movie_id AND t.id = mc.movie_id AND mk.movie_id = mi.movie_id AND mk [...]
diff --git a/benchmarks/queries/imdb/16a.sql b/benchmarks/queries/imdb/16a.sql
new file mode 100644
index 0000000000..aaa0020269
--- /dev/null
+++ b/benchmarks/queries/imdb/16a.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS cool_actor_pseudonym, MIN(t.title) AS series_named_after_char FROM aka_name AS an, cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND t.episode_nr >= 50 AND t.episode_nr < 100 AND an.person_id = n.id AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.com [...]
diff --git a/benchmarks/queries/imdb/16b.sql b/benchmarks/queries/imdb/16b.sql
new file mode 100644
index 0000000000..c6c0bef319
--- /dev/null
+++ b/benchmarks/queries/imdb/16b.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS cool_actor_pseudonym, MIN(t.title) AS series_named_after_char FROM aka_name AS an, cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND an.person_id = n.id AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND an.person_id = ci.person_i [...]
diff --git a/benchmarks/queries/imdb/16c.sql b/benchmarks/queries/imdb/16c.sql
new file mode 100644
index 0000000000..5c3b357521
--- /dev/null
+++ b/benchmarks/queries/imdb/16c.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS cool_actor_pseudonym, MIN(t.title) AS series_named_after_char FROM aka_name AS an, cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND t.episode_nr < 100 AND an.person_id = n.id AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND an. [...]
diff --git a/benchmarks/queries/imdb/16d.sql b/benchmarks/queries/imdb/16d.sql
new file mode 100644
index 0000000000..c9e1b5f25c
--- /dev/null
+++ b/benchmarks/queries/imdb/16d.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS cool_actor_pseudonym, MIN(t.title) AS series_named_after_char FROM aka_name AS an, cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND t.episode_nr >= 5 AND t.episode_nr < 100 AND an.person_id = n.id AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.comp [...]
diff --git a/benchmarks/queries/imdb/17a.sql b/benchmarks/queries/imdb/17a.sql
new file mode 100644
index 0000000000..e854a957e4
--- /dev/null
+++ b/benchmarks/queries/imdb/17a.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_american_movie, MIN(n.name) AS a1 FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND n.name LIKE 'B%' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_ [...]
diff --git a/benchmarks/queries/imdb/17b.sql b/benchmarks/queries/imdb/17b.sql
new file mode 100644
index 0000000000..903f2196b2
--- /dev/null
+++ b/benchmarks/queries/imdb/17b.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_movie, MIN(n.name) AS a1 FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword ='character-name-in-title' AND n.name LIKE 'Z%' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/17c.sql b/benchmarks/queries/imdb/17c.sql
new file mode 100644
index 0000000000..a96faa0b43
--- /dev/null
+++ b/benchmarks/queries/imdb/17c.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_movie, MIN(n.name) AS a1 FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword ='character-name-in-title' AND n.name LIKE 'X%' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/17d.sql b/benchmarks/queries/imdb/17d.sql
new file mode 100644
index 0000000000..73e1f2c309
--- /dev/null
+++ b/benchmarks/queries/imdb/17d.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_movie FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword ='character-name-in-title' AND n.name LIKE '%Bert%' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/17e.sql b/benchmarks/queries/imdb/17e.sql
new file mode 100644
index 0000000000..65ea73ed05
--- /dev/null
+++ b/benchmarks/queries/imdb/17e.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_movie FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/17f.sql b/benchmarks/queries/imdb/17f.sql
new file mode 100644
index 0000000000..542233d63e
--- /dev/null
+++ b/benchmarks/queries/imdb/17f.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS member_in_charnamed_movie FROM cast_info AS ci, company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword ='character-name-in-title' AND n.name LIKE '%B%' AND n.id = ci.person_id AND ci.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.movie_id = mc.movie_id AND ci.movie_id = mk.movie_id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/18a.sql b/benchmarks/queries/imdb/18a.sql
new file mode 100644
index 0000000000..275e04bdb1
--- /dev/null
+++ b/benchmarks/queries/imdb/18a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(t.title) AS movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, movie_info AS mi, movie_info_idx AS mi_idx, name AS n, title AS t WHERE ci.note in ('(producer)', '(executive producer)') AND it1.info = 'budget' AND it2.info = 'votes' AND n.gender = 'm' and n.name like '%Tim%' AND t.id = mi.movie_id AND t.id = mi_idx.movie_id AND t.id = ci.movie_id AND ci.movie_id = mi.movie_id AND ci.movie_id = [...]
diff --git a/benchmarks/queries/imdb/18b.sql b/benchmarks/queries/imdb/18b.sql
new file mode 100644
index 0000000000..3ae40ed93d
--- /dev/null
+++ b/benchmarks/queries/imdb/18b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(t.title) AS movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, movie_info AS mi, movie_info_idx AS mi_idx, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND it1.info = 'genres' AND it2.info = 'rating' AND mi.info in ('Horror', 'Thriller') and mi.note is NULL AND mi_idx.info > '8.0' AND n.gender is not null and n.gender = 'f [...]
diff --git a/benchmarks/queries/imdb/18c.sql b/benchmarks/queries/imdb/18c.sql
new file mode 100644
index 0000000000..01f28ea527
--- /dev/null
+++ b/benchmarks/queries/imdb/18c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(t.title) AS movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, movie_info AS mi, movie_info_idx AS mi_idx, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND it1.info = 'genres' AND it2.info = 'votes' AND mi.info in ('Horror', 'Action', 'Sci-Fi', 'Thriller', 'Crime', 'War') AND n.gender = 'm' AND t.id = mi.movie_id AND t.id = [...]
diff --git a/benchmarks/queries/imdb/19a.sql b/benchmarks/queries/imdb/19a.sql
new file mode 100644
index 0000000000..ceaae671fd
--- /dev/null
+++ b/benchmarks/queries/imdb/19a.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS voicing_actress, MIN(t.title) AS voiced_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, movie_companies AS mc, movie_info AS mi, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND it.info = 'release dates' AND mc.note is not NULL and (mc.note like '%(USA)%' or mc.note like '%(worldwide [...]
diff --git a/benchmarks/queries/imdb/19b.sql b/benchmarks/queries/imdb/19b.sql
new file mode 100644
index 0000000000..62e852ba3e
--- /dev/null
+++ b/benchmarks/queries/imdb/19b.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS voicing_actress, MIN(t.title) AS kung_fu_panda FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, movie_companies AS mc, movie_info AS mi, name AS n, role_type AS rt, title AS t WHERE ci.note = '(voice)' AND cn.country_code ='[us]' AND it.info = 'release dates' AND mc.note like '%(200%)%' and (mc.note like '%(USA)%' or mc.note like '%(worldwide)%') AND mi.info is not null and (mi.info like 'Japan:%2007%' or mi.info like [...]
diff --git a/benchmarks/queries/imdb/19c.sql b/benchmarks/queries/imdb/19c.sql
new file mode 100644
index 0000000000..6885af5012
--- /dev/null
+++ b/benchmarks/queries/imdb/19c.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS voicing_actress, MIN(t.title) AS jap_engl_voiced_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, movie_companies AS mc, movie_info AS mi, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND it.info = 'release dates' AND mi.info is not null and (mi.info like 'Japan:%200%' or mi.info like [...]
diff --git a/benchmarks/queries/imdb/19d.sql b/benchmarks/queries/imdb/19d.sql
new file mode 100644
index 0000000000..06fcc76ba7
--- /dev/null
+++ b/benchmarks/queries/imdb/19d.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS voicing_actress, MIN(t.title) AS jap_engl_voiced_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, movie_companies AS mc, movie_info AS mi, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND it.info = 'release dates' AND n.gender ='f' AND rt.role ='actress' AND t.production_year > 2000 A [...]
diff --git a/benchmarks/queries/imdb/1a.sql b/benchmarks/queries/imdb/1a.sql
new file mode 100644
index 0000000000..07b3516388
--- /dev/null
+++ b/benchmarks/queries/imdb/1a.sql
@@ -0,0 +1 @@
+SELECT MIN(mc.note) AS production_note, MIN(t.title) AS movie_title, MIN(t.production_year) AS movie_year FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info_idx AS mi_idx, title AS t WHERE ct.kind = 'production companies' AND it.info = 'top 250 rank' AND mc.note not like '%(as Metro-Goldwyn-Mayer Pictures)%' and (mc.note like '%(co-production)%' or mc.note like '%(presents)%') AND ct.id = mc.company_type_id AND t.id = mc.movie_id AND t.id = mi_idx.movie_id AND m [...]
diff --git a/benchmarks/queries/imdb/1b.sql b/benchmarks/queries/imdb/1b.sql
new file mode 100644
index 0000000000..f2901e8b52
--- /dev/null
+++ b/benchmarks/queries/imdb/1b.sql
@@ -0,0 +1 @@
+SELECT MIN(mc.note) AS production_note, MIN(t.title) AS movie_title, MIN(t.production_year) AS movie_year FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info_idx AS mi_idx, title AS t WHERE ct.kind = 'production companies' AND it.info = 'bottom 10 rank' AND mc.note not like '%(as Metro-Goldwyn-Mayer Pictures)%' AND t.production_year between 2005 and 2010 AND ct.id = mc.company_type_id AND t.id = mc.movie_id AND t.id = mi_idx.movie_id AND mc.movie_id = mi_idx.movi [...]
diff --git a/benchmarks/queries/imdb/1c.sql b/benchmarks/queries/imdb/1c.sql
new file mode 100644
index 0000000000..94e66c30aa
--- /dev/null
+++ b/benchmarks/queries/imdb/1c.sql
@@ -0,0 +1 @@
+SELECT MIN(mc.note) AS production_note, MIN(t.title) AS movie_title, MIN(t.production_year) AS movie_year FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info_idx AS mi_idx, title AS t WHERE ct.kind = 'production companies' AND it.info = 'top 250 rank' AND mc.note not like '%(as Metro-Goldwyn-Mayer Pictures)%' and (mc.note like '%(co-production)%') AND t.production_year >2010 AND ct.id = mc.company_type_id AND t.id = mc.movie_id AND t.id = mi_idx.movie_id AND mc.m [...]
diff --git a/benchmarks/queries/imdb/1d.sql b/benchmarks/queries/imdb/1d.sql
new file mode 100644
index 0000000000..52f58e80c8
--- /dev/null
+++ b/benchmarks/queries/imdb/1d.sql
@@ -0,0 +1 @@
+SELECT MIN(mc.note) AS production_note, MIN(t.title) AS movie_title, MIN(t.production_year) AS movie_year FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info_idx AS mi_idx, title AS t WHERE ct.kind = 'production companies' AND it.info = 'bottom 10 rank' AND mc.note not like '%(as Metro-Goldwyn-Mayer Pictures)%' AND t.production_year >2000 AND ct.id = mc.company_type_id AND t.id = mc.movie_id AND t.id = mi_idx.movie_id AND mc.movie_id = mi_idx.movie_id AND it.id = [...]
diff --git a/benchmarks/queries/imdb/20a.sql b/benchmarks/queries/imdb/20a.sql
new file mode 100644
index 0000000000..2a1c269d6a
--- /dev/null
+++ b/benchmarks/queries/imdb/20a.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS complete_downey_ironman_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, keyword AS k, kind_type AS kt, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name not like '%Sherlock%' and (chn.name like '%Tony%Stark%' or chn.name like '%Iron%Man%') AND k.keyword in ('superhero', 'sequel', 'second-part', 'marvel-comics', 'based-on-comic', 'tv- [...]
diff --git a/benchmarks/queries/imdb/20b.sql b/benchmarks/queries/imdb/20b.sql
new file mode 100644
index 0000000000..4c2455a52e
--- /dev/null
+++ b/benchmarks/queries/imdb/20b.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS complete_downey_ironman_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, keyword AS k, kind_type AS kt, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name not like '%Sherlock%' and (chn.name like '%Tony%Stark%' or chn.name like '%Iron%Man%') AND k.keyword in ('superhero', 'sequel', 'second-part', 'marvel-comics', 'based-on-comic', 'tv- [...]
diff --git a/benchmarks/queries/imdb/20c.sql b/benchmarks/queries/imdb/20c.sql
new file mode 100644
index 0000000000..b85b22f6b4
--- /dev/null
+++ b/benchmarks/queries/imdb/20c.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS cast_member, MIN(t.title) AS complete_dynamic_hero_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, keyword AS k, kind_type AS kt, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name is not NULL and (chn.name like '%man%' or chn.name like '%Man%') AND k.keyword in ('superhero', 'marvel-comics', 'based-on-comic', 'tv-special', 'fight', 'v [...]
diff --git a/benchmarks/queries/imdb/21a.sql b/benchmarks/queries/imdb/21a.sql
new file mode 100644
index 0000000000..8a66a00be6
--- /dev/null
+++ b/benchmarks/queries/imdb/21a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS company_name, MIN(lt.link) AS link_type, MIN(t.title) AS western_follow_up FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND ct.kind ='production companies' AND k.keyword ='sequel' AND lt.link LIKE '%follow%' AND mc.note IS NULL AND mi.info IN ('Sweden', 'Norway [...]
diff --git a/benchmarks/queries/imdb/21b.sql b/benchmarks/queries/imdb/21b.sql
new file mode 100644
index 0000000000..90d3a5a4c0
--- /dev/null
+++ b/benchmarks/queries/imdb/21b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS company_name, MIN(lt.link) AS link_type, MIN(t.title) AS german_follow_up FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND ct.kind ='production companies' AND k.keyword ='sequel' AND lt.link LIKE '%follow%' AND mc.note IS NULL AND mi.info IN ('Germany', 'German [...]
diff --git a/benchmarks/queries/imdb/21c.sql b/benchmarks/queries/imdb/21c.sql
new file mode 100644
index 0000000000..16a42ae6f4
--- /dev/null
+++ b/benchmarks/queries/imdb/21c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS company_name, MIN(lt.link) AS link_type, MIN(t.title) AS western_follow_up FROM company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND ct.kind ='production companies' AND k.keyword ='sequel' AND lt.link LIKE '%follow%' AND mc.note IS NULL AND mi.info IN ('Sweden', 'Norway [...]
diff --git a/benchmarks/queries/imdb/22a.sql b/benchmarks/queries/imdb/22a.sql
new file mode 100644
index 0000000000..e513799698
--- /dev/null
+++ b/benchmarks/queries/imdb/22a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS western_violent_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cn.country_code != '[us]' AND it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind in ('movi [...]
diff --git a/benchmarks/queries/imdb/22b.sql b/benchmarks/queries/imdb/22b.sql
new file mode 100644
index 0000000000..f98d0ea809
--- /dev/null
+++ b/benchmarks/queries/imdb/22b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS western_violent_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cn.country_code != '[us]' AND it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind in ('movi [...]
diff --git a/benchmarks/queries/imdb/22c.sql b/benchmarks/queries/imdb/22c.sql
new file mode 100644
index 0000000000..cf757956e0
--- /dev/null
+++ b/benchmarks/queries/imdb/22c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS western_violent_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cn.country_code != '[us]' AND it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind in ('movi [...]
diff --git a/benchmarks/queries/imdb/22d.sql b/benchmarks/queries/imdb/22d.sql
new file mode 100644
index 0000000000..a47feeb051
--- /dev/null
+++ b/benchmarks/queries/imdb/22d.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS western_violent_movie FROM company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cn.country_code != '[us]' AND it1.info = 'countries' AND it2.info = 'rating' AND k.keyword in ('murder', 'murder-in-title', 'blood', 'violence') AND kt.kind in ('movi [...]
diff --git a/benchmarks/queries/imdb/23a.sql b/benchmarks/queries/imdb/23a.sql
new file mode 100644
index 0000000000..724da913b5
--- /dev/null
+++ b/benchmarks/queries/imdb/23a.sql
@@ -0,0 +1 @@
+SELECT MIN(kt.kind) AS movie_kind, MIN(t.title) AS complete_us_internet_movie FROM complete_cast AS cc, comp_cast_type AS cct1, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cct1.kind = 'complete+verified' AND cn.country_code = '[us]' AND it1.info = 'release dates' AND kt.kind in ('movie') AND mi.note like '%internet%' AND mi.info is not NULL and (mi.info like ' [...]
diff --git a/benchmarks/queries/imdb/23b.sql b/benchmarks/queries/imdb/23b.sql
new file mode 100644
index 0000000000..e39f0ecc28
--- /dev/null
+++ b/benchmarks/queries/imdb/23b.sql
@@ -0,0 +1 @@
+SELECT MIN(kt.kind) AS movie_kind, MIN(t.title) AS complete_nerdy_internet_movie FROM complete_cast AS cc, comp_cast_type AS cct1, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cct1.kind = 'complete+verified' AND cn.country_code = '[us]' AND it1.info = 'release dates' AND k.keyword in ('nerd', 'loner', 'alienation', 'dignity') AND kt.kind in ('movie') AND mi.note [...]
diff --git a/benchmarks/queries/imdb/23c.sql b/benchmarks/queries/imdb/23c.sql
new file mode 100644
index 0000000000..839d762d05
--- /dev/null
+++ b/benchmarks/queries/imdb/23c.sql
@@ -0,0 +1 @@
+SELECT MIN(kt.kind) AS movie_kind, MIN(t.title) AS complete_us_internet_movie FROM complete_cast AS cc, comp_cast_type AS cct1, company_name AS cn, company_type AS ct, info_type AS it1, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, title AS t WHERE cct1.kind = 'complete+verified' AND cn.country_code = '[us]' AND it1.info = 'release dates' AND kt.kind in ('movie', 'tv movie', 'video movie', 'video game') AND mi.note like '%internet%' AND [...]
diff --git a/benchmarks/queries/imdb/24a.sql b/benchmarks/queries/imdb/24a.sql
new file mode 100644
index 0000000000..8f10621e02
--- /dev/null
+++ b/benchmarks/queries/imdb/24a.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS voiced_char_name, MIN(n.name) AS voicing_actress_name, MIN(t.title) AS voiced_action_movie_jap_eng FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND it.info = 'releas [...]
diff --git a/benchmarks/queries/imdb/24b.sql b/benchmarks/queries/imdb/24b.sql
new file mode 100644
index 0000000000..d8a2836000
--- /dev/null
+++ b/benchmarks/queries/imdb/24b.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS voiced_char_name, MIN(n.name) AS voicing_actress_name, MIN(t.title) AS kung_fu_panda FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND cn.name = 'DreamWorks Animation [...]
diff --git a/benchmarks/queries/imdb/25a.sql b/benchmarks/queries/imdb/25a.sql
new file mode 100644
index 0000000000..bc55cc01d2
--- /dev/null
+++ b/benchmarks/queries/imdb/25a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS male_writer, MIN(t.title) AS violent_movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND it1.info = 'genres' AND it2.info = 'votes' AND k.keyword in ('murder', 'blood', 'gore', 'death', 'f [...]
diff --git a/benchmarks/queries/imdb/25b.sql b/benchmarks/queries/imdb/25b.sql
new file mode 100644
index 0000000000..3457655bb9
--- /dev/null
+++ b/benchmarks/queries/imdb/25b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS male_writer, MIN(t.title) AS violent_movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND it1.info = 'genres' AND it2.info = 'votes' AND k.keyword in ('murder', 'blood', 'gore', 'death', 'f [...]
diff --git a/benchmarks/queries/imdb/25c.sql b/benchmarks/queries/imdb/25c.sql
new file mode 100644
index 0000000000..cf56a313d8
--- /dev/null
+++ b/benchmarks/queries/imdb/25c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS male_writer, MIN(t.title) AS violent_movie_title FROM cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND it1.info = 'genres' AND it2.info = 'votes' AND k.keyword in ('murder', 'violence', 'blood', 'gore', [...]
diff --git a/benchmarks/queries/imdb/26a.sql b/benchmarks/queries/imdb/26a.sql
new file mode 100644
index 0000000000..b431f204c6
--- /dev/null
+++ b/benchmarks/queries/imdb/26a.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS character_name, MIN(mi_idx.info) AS rating, MIN(n.name) AS playing_actor, MIN(t.title) AS complete_hero_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, info_type AS it2, keyword AS k, kind_type AS kt, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name is not NULL and (chn.name like '%man%' or chn.name like '% [...]
diff --git a/benchmarks/queries/imdb/26b.sql b/benchmarks/queries/imdb/26b.sql
new file mode 100644
index 0000000000..882d234d77
--- /dev/null
+++ b/benchmarks/queries/imdb/26b.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS character_name, MIN(mi_idx.info) AS rating, MIN(t.title) AS complete_hero_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, info_type AS it2, keyword AS k, kind_type AS kt, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name is not NULL and (chn.name like '%man%' or chn.name like '%Man%') AND it2.info = 'rating [...]
diff --git a/benchmarks/queries/imdb/26c.sql b/benchmarks/queries/imdb/26c.sql
new file mode 100644
index 0000000000..4b9eae0b76
--- /dev/null
+++ b/benchmarks/queries/imdb/26c.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS character_name, MIN(mi_idx.info) AS rating, MIN(t.title) AS complete_hero_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, info_type AS it2, keyword AS k, kind_type AS kt, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like '%complete%' AND chn.name is not NULL and (chn.name like '%man%' or chn.name like '%Man%') AND it2.info = 'rating [...]
diff --git a/benchmarks/queries/imdb/27a.sql b/benchmarks/queries/imdb/27a.sql
new file mode 100644
index 0000000000..239673cd81
--- /dev/null
+++ b/benchmarks/queries/imdb/27a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(lt.link) AS link_type, MIN(t.title) AS complete_western_sequel FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cct1.kind in ('cast', 'crew') AND cct2.kind = 'complete' AND cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%' [...]
diff --git a/benchmarks/queries/imdb/27b.sql b/benchmarks/queries/imdb/27b.sql
new file mode 100644
index 0000000000..4bf85260f2
--- /dev/null
+++ b/benchmarks/queries/imdb/27b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(lt.link) AS link_type, MIN(t.title) AS complete_western_sequel FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cct1.kind in ('cast', 'crew') AND cct2.kind = 'complete' AND cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%' [...]
diff --git a/benchmarks/queries/imdb/27c.sql b/benchmarks/queries/imdb/27c.sql
new file mode 100644
index 0000000000..dc26ebff68
--- /dev/null
+++ b/benchmarks/queries/imdb/27c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS producing_company, MIN(lt.link) AS link_type, MIN(t.title) AS complete_western_sequel FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, keyword AS k, link_type AS lt, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, movie_link AS ml, title AS t WHERE cct1.kind = 'cast' AND cct2.kind like 'complete%' AND cn.country_code !='[pl]' AND (cn.name LIKE '%Film%' OR cn.name LIKE '%Warner%') AND c [...]
diff --git a/benchmarks/queries/imdb/28a.sql b/benchmarks/queries/imdb/28a.sql
new file mode 100644
index 0000000000..8cb1177386
--- /dev/null
+++ b/benchmarks/queries/imdb/28a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS complete_euro_dark_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cct1.kind = 'crew' AND cct2.kind != 'complete+verified' AND cn.country_code != '[us]' AND it1.in [...]
diff --git a/benchmarks/queries/imdb/28b.sql b/benchmarks/queries/imdb/28b.sql
new file mode 100644
index 0000000000..10f43c8982
--- /dev/null
+++ b/benchmarks/queries/imdb/28b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS complete_euro_dark_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cct1.kind = 'crew' AND cct2.kind != 'complete+verified' AND cn.country_code != '[us]' AND it1.in [...]
diff --git a/benchmarks/queries/imdb/28c.sql b/benchmarks/queries/imdb/28c.sql
new file mode 100644
index 0000000000..6b2e4047ae
--- /dev/null
+++ b/benchmarks/queries/imdb/28c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn.name) AS movie_company, MIN(mi_idx.info) AS rating, MIN(t.title) AS complete_euro_dark_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, company_name AS cn, company_type AS ct, info_type AS it1, info_type AS it2, keyword AS k, kind_type AS kt, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE cct1.kind = 'cast' AND cct2.kind = 'complete' AND cn.country_code != '[us]' AND it1.info = 'cou [...]
diff --git a/benchmarks/queries/imdb/29a.sql b/benchmarks/queries/imdb/29a.sql
new file mode 100644
index 0000000000..3033acbe6c
--- /dev/null
+++ b/benchmarks/queries/imdb/29a.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS voiced_char, MIN(n.name) AS voicing_actress, MIN(t.title) AS voiced_animation FROM aka_name AS an, complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, info_type AS it3, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, name AS n, person_info AS pi, role_type AS rt, title AS t WHERE cct1.kind ='cast' AND cct2.kind ='complete+verified' AND chn.name = [...]
diff --git a/benchmarks/queries/imdb/29b.sql b/benchmarks/queries/imdb/29b.sql
new file mode 100644
index 0000000000..88d50fc7b7
--- /dev/null
+++ b/benchmarks/queries/imdb/29b.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS voiced_char, MIN(n.name) AS voicing_actress, MIN(t.title) AS voiced_animation FROM aka_name AS an, complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, info_type AS it3, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, name AS n, person_info AS pi, role_type AS rt, title AS t WHERE cct1.kind ='cast' AND cct2.kind ='complete+verified' AND chn.name = [...]
diff --git a/benchmarks/queries/imdb/29c.sql b/benchmarks/queries/imdb/29c.sql
new file mode 100644
index 0000000000..cb95178182
--- /dev/null
+++ b/benchmarks/queries/imdb/29c.sql
@@ -0,0 +1 @@
+SELECT MIN(chn.name) AS voiced_char, MIN(n.name) AS voicing_actress, MIN(t.title) AS voiced_animation FROM aka_name AS an, complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, char_name AS chn, cast_info AS ci, company_name AS cn, info_type AS it, info_type AS it3, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_keyword AS mk, name AS n, person_info AS pi, role_type AS rt, title AS t WHERE cct1.kind ='cast' AND cct2.kind ='complete+verified' AND ci.note in [...]
diff --git a/benchmarks/queries/imdb/2a.sql b/benchmarks/queries/imdb/2a.sql
new file mode 100644
index 0000000000..f3ef4db75f
--- /dev/null
+++ b/benchmarks/queries/imdb/2a.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, title AS t WHERE cn.country_code ='[de]' AND k.keyword ='character-name-in-title' AND cn.id = mc.company_id AND mc.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/2b.sql b/benchmarks/queries/imdb/2b.sql
new file mode 100644
index 0000000000..82b2123fbc
--- /dev/null
+++ b/benchmarks/queries/imdb/2b.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, title AS t WHERE cn.country_code ='[nl]' AND k.keyword ='character-name-in-title' AND cn.id = mc.company_id AND mc.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/2c.sql b/benchmarks/queries/imdb/2c.sql
new file mode 100644
index 0000000000..b5f9b75dd6
--- /dev/null
+++ b/benchmarks/queries/imdb/2c.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, title AS t WHERE cn.country_code ='[sm]' AND k.keyword ='character-name-in-title' AND cn.id = mc.company_id AND mc.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/2d.sql b/benchmarks/queries/imdb/2d.sql
new file mode 100644
index 0000000000..4a27919465
--- /dev/null
+++ b/benchmarks/queries/imdb/2d.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM company_name AS cn, keyword AS k, movie_companies AS mc, movie_keyword AS mk, title AS t WHERE cn.country_code ='[us]' AND k.keyword ='character-name-in-title' AND cn.id = mc.company_id AND mc.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND mc.movie_id = mk.movie_id;
diff --git a/benchmarks/queries/imdb/30a.sql b/benchmarks/queries/imdb/30a.sql
new file mode 100644
index 0000000000..698872fa83
--- /dev/null
+++ b/benchmarks/queries/imdb/30a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS complete_violent_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind in ('cast', 'crew') AND cct2.kind ='complete+verified' AND ci.note in ('(writer)', '(head writer)', '(written by) [...]
diff --git a/benchmarks/queries/imdb/30b.sql b/benchmarks/queries/imdb/30b.sql
new file mode 100644
index 0000000000..5fdb849349
--- /dev/null
+++ b/benchmarks/queries/imdb/30b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS complete_gore_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind in ('cast', 'crew') AND cct2.kind ='complete+verified' AND ci.note in ('(writer)', '(head writer)', '(written by)', [...]
diff --git a/benchmarks/queries/imdb/30c.sql b/benchmarks/queries/imdb/30c.sql
new file mode 100644
index 0000000000..a18087e392
--- /dev/null
+++ b/benchmarks/queries/imdb/30c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS complete_violent_movie FROM complete_cast AS cc, comp_cast_type AS cct1, comp_cast_type AS cct2, cast_info AS ci, info_type AS it1, info_type AS it2, keyword AS k, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE cct1.kind = 'cast' AND cct2.kind ='complete+verified' AND ci.note in ('(writer)', '(head writer)', '(written by)', '(story) [...]
diff --git a/benchmarks/queries/imdb/31a.sql b/benchmarks/queries/imdb/31a.sql
new file mode 100644
index 0000000000..7dd855011f
--- /dev/null
+++ b/benchmarks/queries/imdb/31a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS violent_liongate_movie FROM cast_info AS ci, company_name AS cn, info_type AS it1, info_type AS it2, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND cn.name like 'Lionsgate%' AND it1.info = 'genres' AND it2.i [...]
diff --git a/benchmarks/queries/imdb/31b.sql b/benchmarks/queries/imdb/31b.sql
new file mode 100644
index 0000000000..3be5680f7d
--- /dev/null
+++ b/benchmarks/queries/imdb/31b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS violent_liongate_movie FROM cast_info AS ci, company_name AS cn, info_type AS it1, info_type AS it2, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND cn.name like 'Lionsgate%' AND it1.info = 'genres' AND it2.i [...]
diff --git a/benchmarks/queries/imdb/31c.sql b/benchmarks/queries/imdb/31c.sql
new file mode 100644
index 0000000000..156ea2d5ee
--- /dev/null
+++ b/benchmarks/queries/imdb/31c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi.info) AS movie_budget, MIN(mi_idx.info) AS movie_votes, MIN(n.name) AS writer, MIN(t.title) AS violent_liongate_movie FROM cast_info AS ci, company_name AS cn, info_type AS it1, info_type AS it2, keyword AS k, movie_companies AS mc, movie_info AS mi, movie_info_idx AS mi_idx, movie_keyword AS mk, name AS n, title AS t WHERE ci.note in ('(writer)', '(head writer)', '(written by)', '(story)', '(story editor)') AND cn.name like 'Lionsgate%' AND it1.info = 'genres' AND it2.i [...]
diff --git a/benchmarks/queries/imdb/32a.sql b/benchmarks/queries/imdb/32a.sql
new file mode 100644
index 0000000000..9647fb7106
--- /dev/null
+++ b/benchmarks/queries/imdb/32a.sql
@@ -0,0 +1 @@
+SELECT MIN(lt.link) AS link_type, MIN(t1.title) AS first_movie, MIN(t2.title) AS second_movie FROM keyword AS k, link_type AS lt, movie_keyword AS mk, movie_link AS ml, title AS t1, title AS t2 WHERE k.keyword ='10,000-mile-club' AND mk.keyword_id = k.id AND t1.id = mk.movie_id AND ml.movie_id = t1.id AND ml.linked_movie_id = t2.id AND lt.id = ml.link_type_id AND mk.movie_id = t1.id;
diff --git a/benchmarks/queries/imdb/32b.sql b/benchmarks/queries/imdb/32b.sql
new file mode 100644
index 0000000000..6d096ab434
--- /dev/null
+++ b/benchmarks/queries/imdb/32b.sql
@@ -0,0 +1 @@
+SELECT MIN(lt.link) AS link_type, MIN(t1.title) AS first_movie, MIN(t2.title) AS second_movie FROM keyword AS k, link_type AS lt, movie_keyword AS mk, movie_link AS ml, title AS t1, title AS t2 WHERE k.keyword ='character-name-in-title' AND mk.keyword_id = k.id AND t1.id = mk.movie_id AND ml.movie_id = t1.id AND ml.linked_movie_id = t2.id AND lt.id = ml.link_type_id AND mk.movie_id = t1.id;
diff --git a/benchmarks/queries/imdb/33a.sql b/benchmarks/queries/imdb/33a.sql
new file mode 100644
index 0000000000..24aac4e207
--- /dev/null
+++ b/benchmarks/queries/imdb/33a.sql
@@ -0,0 +1 @@
+SELECT MIN(cn1.name) AS first_company, MIN(cn2.name) AS second_company, MIN(mi_idx1.info) AS first_rating, MIN(mi_idx2.info) AS second_rating, MIN(t1.title) AS first_movie, MIN(t2.title) AS second_movie FROM company_name AS cn1, company_name AS cn2, info_type AS it1, info_type AS it2, kind_type AS kt1, kind_type AS kt2, link_type AS lt, movie_companies AS mc1, movie_companies AS mc2, movie_info_idx AS mi_idx1, movie_info_idx AS mi_idx2, movie_link AS ml, title AS t1, title AS t2 WHERE cn [...]
diff --git a/benchmarks/queries/imdb/33b.sql b/benchmarks/queries/imdb/33b.sql
new file mode 100644
index 0000000000..fe6fd75a69
--- /dev/null
+++ b/benchmarks/queries/imdb/33b.sql
@@ -0,0 +1 @@
+SELECT MIN(cn1.name) AS first_company, MIN(cn2.name) AS second_company, MIN(mi_idx1.info) AS first_rating, MIN(mi_idx2.info) AS second_rating, MIN(t1.title) AS first_movie, MIN(t2.title) AS second_movie FROM company_name AS cn1, company_name AS cn2, info_type AS it1, info_type AS it2, kind_type AS kt1, kind_type AS kt2, link_type AS lt, movie_companies AS mc1, movie_companies AS mc2, movie_info_idx AS mi_idx1, movie_info_idx AS mi_idx2, movie_link AS ml, title AS t1, title AS t2 WHERE cn [...]
diff --git a/benchmarks/queries/imdb/33c.sql b/benchmarks/queries/imdb/33c.sql
new file mode 100644
index 0000000000..c9f0907d3f
--- /dev/null
+++ b/benchmarks/queries/imdb/33c.sql
@@ -0,0 +1 @@
+SELECT MIN(cn1.name) AS first_company, MIN(cn2.name) AS second_company, MIN(mi_idx1.info) AS first_rating, MIN(mi_idx2.info) AS second_rating, MIN(t1.title) AS first_movie, MIN(t2.title) AS second_movie FROM company_name AS cn1, company_name AS cn2, info_type AS it1, info_type AS it2, kind_type AS kt1, kind_type AS kt2, link_type AS lt, movie_companies AS mc1, movie_companies AS mc2, movie_info_idx AS mi_idx1, movie_info_idx AS mi_idx2, movie_link AS ml, title AS t1, title AS t2 WHERE cn [...]
diff --git a/benchmarks/queries/imdb/3a.sql b/benchmarks/queries/imdb/3a.sql
new file mode 100644
index 0000000000..231c957be2
--- /dev/null
+++ b/benchmarks/queries/imdb/3a.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM keyword AS k, movie_info AS mi, movie_keyword AS mk, title AS t WHERE k.keyword like '%sequel%' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German') AND t.production_year > 2005 AND t.id = mi.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi.movie_id AND k.id = mk.keyword_id;
diff --git a/benchmarks/queries/imdb/3b.sql b/benchmarks/queries/imdb/3b.sql
new file mode 100644
index 0000000000..fd21efc810
--- /dev/null
+++ b/benchmarks/queries/imdb/3b.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM keyword AS k, movie_info AS mi, movie_keyword AS mk, title AS t WHERE k.keyword like '%sequel%' AND mi.info IN ('Bulgaria') AND t.production_year > 2010 AND t.id = mi.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi.movie_id AND k.id = mk.keyword_id;
diff --git a/benchmarks/queries/imdb/3c.sql b/benchmarks/queries/imdb/3c.sql
new file mode 100644
index 0000000000..5f34232a2e
--- /dev/null
+++ b/benchmarks/queries/imdb/3c.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS movie_title FROM keyword AS k, movie_info AS mi, movie_keyword AS mk, title AS t WHERE k.keyword like '%sequel%' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German', 'USA', 'American') AND t.production_year > 1990 AND t.id = mi.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi.movie_id AND k.id = mk.keyword_id;
diff --git a/benchmarks/queries/imdb/4a.sql b/benchmarks/queries/imdb/4a.sql
new file mode 100644
index 0000000000..636afab02c
--- /dev/null
+++ b/benchmarks/queries/imdb/4a.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS movie_title FROM info_type AS it, keyword AS k, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it.info ='rating' AND k.keyword like '%sequel%' AND mi_idx.info > '5.0' AND t.production_year > 2005 AND t.id = mi_idx.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi_idx.movie_id AND k.id = mk.keyword_id AND it.id = mi_idx.info_type_id;
diff --git a/benchmarks/queries/imdb/4b.sql b/benchmarks/queries/imdb/4b.sql
new file mode 100644
index 0000000000..ebd3e89920
--- /dev/null
+++ b/benchmarks/queries/imdb/4b.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS movie_title FROM info_type AS it, keyword AS k, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it.info ='rating' AND k.keyword like '%sequel%' AND mi_idx.info > '9.0' AND t.production_year > 2010 AND t.id = mi_idx.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi_idx.movie_id AND k.id = mk.keyword_id AND it.id = mi_idx.info_type_id;
diff --git a/benchmarks/queries/imdb/4c.sql b/benchmarks/queries/imdb/4c.sql
new file mode 100644
index 0000000000..309281200f
--- /dev/null
+++ b/benchmarks/queries/imdb/4c.sql
@@ -0,0 +1 @@
+SELECT MIN(mi_idx.info) AS rating, MIN(t.title) AS movie_title FROM info_type AS it, keyword AS k, movie_info_idx AS mi_idx, movie_keyword AS mk, title AS t WHERE it.info ='rating' AND k.keyword like '%sequel%' AND mi_idx.info > '2.0' AND t.production_year > 1990 AND t.id = mi_idx.movie_id AND t.id = mk.movie_id AND mk.movie_id = mi_idx.movie_id AND k.id = mk.keyword_id AND it.id = mi_idx.info_type_id;
diff --git a/benchmarks/queries/imdb/5a.sql b/benchmarks/queries/imdb/5a.sql
new file mode 100644
index 0000000000..04aae9881f
--- /dev/null
+++ b/benchmarks/queries/imdb/5a.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS typical_european_movie FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info AS mi, title AS t WHERE ct.kind = 'production companies' AND mc.note like '%(theatrical)%' and mc.note like '%(France)%' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German') AND t.production_year > 2005 AND t.id = mi.movie_id AND t.id = mc.movie_id AND mc.movie_id = mi.movie_id AND ct.id = mc.company_type_id AND it.i [...]
diff --git a/benchmarks/queries/imdb/5b.sql b/benchmarks/queries/imdb/5b.sql
new file mode 100644
index 0000000000..f03a519d61
--- /dev/null
+++ b/benchmarks/queries/imdb/5b.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS american_vhs_movie FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info AS mi, title AS t WHERE ct.kind = 'production companies' AND mc.note like '%(VHS)%' and mc.note like '%(USA)%' and mc.note like '%(1994)%' AND mi.info IN ('USA', 'America') AND t.production_year > 2010 AND t.id = mi.movie_id AND t.id = mc.movie_id AND mc.movie_id = mi.movie_id AND ct.id = mc.company_type_id AND it.id = mi.info_type_id;
diff --git a/benchmarks/queries/imdb/5c.sql b/benchmarks/queries/imdb/5c.sql
new file mode 100644
index 0000000000..2705e7e2c7
--- /dev/null
+++ b/benchmarks/queries/imdb/5c.sql
@@ -0,0 +1 @@
+SELECT MIN(t.title) AS american_movie FROM company_type AS ct, info_type AS it, movie_companies AS mc, movie_info AS mi, title AS t WHERE ct.kind = 'production companies' AND mc.note not like '%(TV)%' and mc.note like '%(USA)%' AND mi.info IN ('Sweden', 'Norway', 'Germany', 'Denmark', 'Swedish', 'Denish', 'Norwegian', 'German', 'USA', 'American') AND t.production_year > 1990 AND t.id = mi.movie_id AND t.id = mc.movie_id AND mc.movie_id = mi.movie_id AND ct.id = mc.company_type_id AND [...]
diff --git a/benchmarks/queries/imdb/6a.sql b/benchmarks/queries/imdb/6a.sql
new file mode 100644
index 0000000000..34b3a6da5f
--- /dev/null
+++ b/benchmarks/queries/imdb/6a.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS marvel_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword = 'marvel-cinematic-universe' AND n.name LIKE '%Downey%Robert%' AND t.production_year > 2010 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/6b.sql b/benchmarks/queries/imdb/6b.sql
new file mode 100644
index 0000000000..1233c41e66
--- /dev/null
+++ b/benchmarks/queries/imdb/6b.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS hero_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword in ('superhero', 'sequel', 'second-part', 'marvel-comics', 'based-on-comic', 'tv-special', 'fight', 'violence') AND n.name LIKE '%Downey%Robert%' AND t.production_year > 2014 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/6c.sql b/benchmarks/queries/imdb/6c.sql
new file mode 100644
index 0000000000..d1f97746e1
--- /dev/null
+++ b/benchmarks/queries/imdb/6c.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS marvel_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword = 'marvel-cinematic-universe' AND n.name LIKE '%Downey%Robert%' AND t.production_year > 2014 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/6d.sql b/benchmarks/queries/imdb/6d.sql
new file mode 100644
index 0000000000..07729510a4
--- /dev/null
+++ b/benchmarks/queries/imdb/6d.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS hero_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword in ('superhero', 'sequel', 'second-part', 'marvel-comics', 'based-on-comic', 'tv-special', 'fight', 'violence') AND n.name LIKE '%Downey%Robert%' AND t.production_year > 2000 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/6e.sql b/benchmarks/queries/imdb/6e.sql
new file mode 100644
index 0000000000..2e77873fd8
--- /dev/null
+++ b/benchmarks/queries/imdb/6e.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS marvel_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword = 'marvel-cinematic-universe' AND n.name LIKE '%Downey%Robert%' AND t.production_year > 2000 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/6f.sql b/benchmarks/queries/imdb/6f.sql
new file mode 100644
index 0000000000..6039011291
--- /dev/null
+++ b/benchmarks/queries/imdb/6f.sql
@@ -0,0 +1 @@
+SELECT MIN(k.keyword) AS movie_keyword, MIN(n.name) AS actor_name, MIN(t.title) AS hero_movie FROM cast_info AS ci, keyword AS k, movie_keyword AS mk, name AS n, title AS t WHERE k.keyword in ('superhero', 'sequel', 'second-part', 'marvel-comics', 'based-on-comic', 'tv-special', 'fight', 'violence') AND t.production_year > 2000 AND k.id = mk.keyword_id AND t.id = mk.movie_id AND t.id = ci.movie_id AND ci.movie_id = mk.movie_id AND n.id = ci.person_id;
diff --git a/benchmarks/queries/imdb/7a.sql b/benchmarks/queries/imdb/7a.sql
new file mode 100644
index 0000000000..c6b26ce36f
--- /dev/null
+++ b/benchmarks/queries/imdb/7a.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS of_person, MIN(t.title) AS biography_movie FROM aka_name AS an, cast_info AS ci, info_type AS it, link_type AS lt, movie_link AS ml, name AS n, person_info AS pi, title AS t WHERE an.name LIKE '%a%' AND it.info ='mini biography' AND lt.link ='features' AND n.name_pcode_cf BETWEEN 'A' AND 'F' AND (n.gender='m' OR (n.gender = 'f' AND n.name LIKE 'B%')) AND pi.note ='Volker Boehm' AND t.production_year BETWEEN 1980 AND 1995 AND n.id = an.person_id AND n.id = pi.person_ [...]
diff --git a/benchmarks/queries/imdb/7b.sql b/benchmarks/queries/imdb/7b.sql
new file mode 100644
index 0000000000..4e4f6e7615
--- /dev/null
+++ b/benchmarks/queries/imdb/7b.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS of_person, MIN(t.title) AS biography_movie FROM aka_name AS an, cast_info AS ci, info_type AS it, link_type AS lt, movie_link AS ml, name AS n, person_info AS pi, title AS t WHERE an.name LIKE '%a%' AND it.info ='mini biography' AND lt.link ='features' AND n.name_pcode_cf LIKE 'D%' AND n.gender='m' AND pi.note ='Volker Boehm' AND t.production_year BETWEEN 1980 AND 1984 AND n.id = an.person_id AND n.id = pi.person_id AND ci.person_id = n.id AND t.id = ci.movie_id AND [...]
diff --git a/benchmarks/queries/imdb/7c.sql b/benchmarks/queries/imdb/7c.sql
new file mode 100644
index 0000000000..a399342fae
--- /dev/null
+++ b/benchmarks/queries/imdb/7c.sql
@@ -0,0 +1 @@
+SELECT MIN(n.name) AS cast_member_name, MIN(pi.info) AS cast_member_info FROM aka_name AS an, cast_info AS ci, info_type AS it, link_type AS lt, movie_link AS ml, name AS n, person_info AS pi, title AS t WHERE an.name is not NULL and (an.name LIKE '%a%' or an.name LIKE 'A%') AND it.info ='mini biography' AND lt.link in ('references', 'referenced in', 'features', 'featured in') AND n.name_pcode_cf BETWEEN 'A' AND 'F' AND (n.gender='m' OR (n.gender = 'f' AND n.name LIKE 'A%')) AND pi.not [...]
diff --git a/benchmarks/queries/imdb/8a.sql b/benchmarks/queries/imdb/8a.sql
new file mode 100644
index 0000000000..66ed05880d
--- /dev/null
+++ b/benchmarks/queries/imdb/8a.sql
@@ -0,0 +1 @@
+SELECT MIN(an1.name) AS actress_pseudonym, MIN(t.title) AS japanese_movie_dubbed FROM aka_name AS an1, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n1, role_type AS rt, title AS t WHERE ci.note ='(voice: English version)' AND cn.country_code ='[jp]' AND mc.note like '%(Japan)%' and mc.note not like '%(USA)%' AND n1.name like '%Yo%' and n1.name not like '%Yu%' AND rt.role ='actress' AND an1.person_id = n1.id AND n1.id = ci.person_id AND ci.movie_id = t.id AND t.id = [...]
diff --git a/benchmarks/queries/imdb/8b.sql b/benchmarks/queries/imdb/8b.sql
new file mode 100644
index 0000000000..044b5f8e86
--- /dev/null
+++ b/benchmarks/queries/imdb/8b.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS acress_pseudonym, MIN(t.title) AS japanese_anime_movie FROM aka_name AS an, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n, role_type AS rt, title AS t WHERE ci.note ='(voice: English version)' AND cn.country_code ='[jp]' AND mc.note like '%(Japan)%' and mc.note not like '%(USA)%' and (mc.note like '%(2006)%' or mc.note like '%(2007)%') AND n.name like '%Yo%' and n.name not like '%Yu%' AND rt.role ='actress' AND t.production_year between 2006 [...]
diff --git a/benchmarks/queries/imdb/8c.sql b/benchmarks/queries/imdb/8c.sql
new file mode 100644
index 0000000000..d02b74c02c
--- /dev/null
+++ b/benchmarks/queries/imdb/8c.sql
@@ -0,0 +1 @@
+SELECT MIN(a1.name) AS writer_pseudo_name, MIN(t.title) AS movie_title FROM aka_name AS a1, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n1, role_type AS rt, title AS t WHERE cn.country_code ='[us]' AND rt.role ='writer' AND a1.person_id = n1.id AND n1.id = ci.person_id AND ci.movie_id = t.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.role_id = rt.id AND a1.person_id = ci.person_id AND ci.movie_id = mc.movie_id;
diff --git a/benchmarks/queries/imdb/8d.sql b/benchmarks/queries/imdb/8d.sql
new file mode 100644
index 0000000000..0834c0ff5c
--- /dev/null
+++ b/benchmarks/queries/imdb/8d.sql
@@ -0,0 +1 @@
+SELECT MIN(an1.name) AS costume_designer_pseudo, MIN(t.title) AS movie_with_costumes FROM aka_name AS an1, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n1, role_type AS rt, title AS t WHERE cn.country_code ='[us]' AND rt.role ='costume designer' AND an1.person_id = n1.id AND n1.id = ci.person_id AND ci.movie_id = t.id AND t.id = mc.movie_id AND mc.company_id = cn.id AND ci.role_id = rt.id AND an1.person_id = ci.person_id AND ci.movie_id = mc.movie_id;
diff --git a/benchmarks/queries/imdb/9a.sql b/benchmarks/queries/imdb/9a.sql
new file mode 100644
index 0000000000..593b16213b
--- /dev/null
+++ b/benchmarks/queries/imdb/9a.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS alternative_name, MIN(chn.name) AS character_name, MIN(t.title) AS movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND mc.note is not NULL and (mc.note like '%(USA)%' or mc.note like '%(worldwide)%') AND n.gender ='f' and n.name like [...]
diff --git a/benchmarks/queries/imdb/9b.sql b/benchmarks/queries/imdb/9b.sql
new file mode 100644
index 0000000000..a4933fd685
--- /dev/null
+++ b/benchmarks/queries/imdb/9b.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS alternative_name, MIN(chn.name) AS voiced_character, MIN(n.name) AS voicing_actress, MIN(t.title) AS american_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n, role_type AS rt, title AS t WHERE ci.note = '(voice)' AND cn.country_code ='[us]' AND mc.note like '%(200%)%' and (mc.note like '%(USA)%' or mc.note like '%(worldwide)%') AND n.gender ='f' and n.name like '%Angel%' AND rt.role ='actress' AND [...]
diff --git a/benchmarks/queries/imdb/9c.sql b/benchmarks/queries/imdb/9c.sql
new file mode 100644
index 0000000000..0be511810c
--- /dev/null
+++ b/benchmarks/queries/imdb/9c.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS alternative_name, MIN(chn.name) AS voiced_character_name, MIN(n.name) AS voicing_actress, MIN(t.title) AS american_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND n.gender ='f' and n.name like '%An%' AND rt.role ='actress' AND ci. [...]
diff --git a/benchmarks/queries/imdb/9d.sql b/benchmarks/queries/imdb/9d.sql
new file mode 100644
index 0000000000..51262ca5eb
--- /dev/null
+++ b/benchmarks/queries/imdb/9d.sql
@@ -0,0 +1 @@
+SELECT MIN(an.name) AS alternative_name, MIN(chn.name) AS voiced_char_name, MIN(n.name) AS voicing_actress, MIN(t.title) AS american_movie FROM aka_name AS an, char_name AS chn, cast_info AS ci, company_name AS cn, movie_companies AS mc, name AS n, role_type AS rt, title AS t WHERE ci.note in ('(voice)', '(voice: Japanese version)', '(voice) (uncredited)', '(voice: English version)') AND cn.country_code ='[us]' AND n.gender ='f' AND rt.role ='actress' AND ci.movie_id = t.id AND t.id = m [...]
diff --git a/benchmarks/src/bin/dfbench.rs b/benchmarks/src/bin/dfbench.rs
index 9ce6848a06..f7b84116e7 100644
--- a/benchmarks/src/bin/dfbench.rs
+++ b/benchmarks/src/bin/dfbench.rs
@@ -33,7 +33,7 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;
#[global_allocator]
static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
-use datafusion_benchmarks::{clickbench, parquet_filter, sort, tpch};
+use datafusion_benchmarks::{clickbench, imdb, parquet_filter, sort, tpch};
#[derive(Debug, StructOpt)]
#[structopt(about = "benchmark command")]
@@ -43,6 +43,7 @@ enum Options {
Clickbench(clickbench::RunOpt),
ParquetFilter(parquet_filter::RunOpt),
Sort(sort::RunOpt),
+ Imdb(imdb::RunOpt),
}
// Main benchmark runner entrypoint
@@ -56,5 +57,6 @@ pub async fn main() -> Result<()> {
Options::Clickbench(opt) => opt.run().await,
Options::ParquetFilter(opt) => opt.run().await,
Options::Sort(opt) => opt.run().await,
+ Options::Imdb(opt) => opt.run().await,
}
}
diff --git a/benchmarks/src/bin/imdb.rs b/benchmarks/src/bin/imdb.rs
index 40efb84b05..13421f8a89 100644
--- a/benchmarks/src/bin/imdb.rs
+++ b/benchmarks/src/bin/imdb.rs
@@ -34,9 +34,17 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;
#[global_allocator]
static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
+#[derive(Debug, StructOpt)]
+#[structopt(about = "benchmark command")]
+enum BenchmarkSubCommandOpt {
+ #[structopt(name = "datafusion")]
+ DataFusionBenchmark(imdb::RunOpt),
+}
+
#[derive(Debug, StructOpt)]
#[structopt(name = "IMDB", about = "IMDB Dataset Processing.")]
enum ImdbOpt {
+ Benchmark(BenchmarkSubCommandOpt),
Convert(imdb::ConvertOpt),
}
@@ -44,6 +52,9 @@ enum ImdbOpt {
pub async fn main() -> Result<()> {
env_logger::init();
match ImdbOpt::from_args() {
+ ImdbOpt::Benchmark(BenchmarkSubCommandOpt::DataFusionBenchmark(opt)) => {
+ opt.run().await
+ }
ImdbOpt::Convert(opt) => opt.run().await,
}
}
diff --git a/benchmarks/src/imdb/convert.rs b/benchmarks/src/imdb/convert.rs
index c95f7f8bf5..4e470d711d 100644
--- a/benchmarks/src/imdb/convert.rs
+++ b/benchmarks/src/imdb/convert.rs
@@ -51,11 +51,12 @@ impl ConvertOpt {
pub async fn run(self) -> Result<()> {
let input_path = self.input_path.to_str().unwrap();
let output_path = self.output_path.to_str().unwrap();
+ let config = SessionConfig::new().with_batch_size(self.batch_size);
+ let ctx = SessionContext::new_with_config(config);
for table in IMDB_TABLES {
let start = Instant::now();
let schema = get_imdb_table_schema(table);
-
let input_path = format!("{input_path}/{table}.csv");
let output_path = format!("{output_path}/{table}.parquet");
let options = CsvReadOptions::new()
@@ -65,9 +66,6 @@ impl ConvertOpt {
.escape(b'\\')
.file_extension(".csv");
- let config = SessionConfig::new().with_batch_size(self.batch_size);
- let ctx = SessionContext::new_with_config(config);
-
let mut csv = ctx.read_csv(&input_path, options).await?;
// Select all apart from the padding column
diff --git a/benchmarks/src/imdb/mod.rs b/benchmarks/src/imdb/mod.rs
index 8e2977c038..6a45242e6f 100644
--- a/benchmarks/src/imdb/mod.rs
+++ b/benchmarks/src/imdb/mod.rs
@@ -17,10 +17,18 @@
//! Benchmark derived from IMDB dataset.
-use datafusion::arrow::datatypes::{DataType, Field, Schema};
+use datafusion::{
+ arrow::datatypes::{DataType, Field, Schema},
+ common::plan_err,
+ error::Result,
+};
mod convert;
pub use convert::ConvertOpt;
+use std::fs;
+mod run;
+pub use run::RunOpt;
+
// we have 21 tables in the IMDB dataset
pub const IMDB_TABLES: &[&str] = &[
"aka_name",
@@ -51,7 +59,7 @@ pub const IMDB_TABLES: &[&str] = &[
pub fn get_imdb_table_schema(table: &str) -> Schema {
match table {
"aka_name" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("person_id", DataType::Int32, false),
Field::new("name", DataType::Utf8, true),
Field::new("imdb_index", DataType::Utf8, true),
@@ -61,7 +69,7 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"aka_title" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("title", DataType::Utf8, true),
Field::new("imdb_index", DataType::Utf8, true),
@@ -75,7 +83,7 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"cast_info" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("person_id", DataType::Int32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("person_role_id", DataType::Int32, true),
@@ -84,7 +92,7 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("role_id", DataType::Int32, false),
]),
"char_name" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("name", DataType::Utf8, false),
Field::new("imdb_index", DataType::Utf8, true),
Field::new("imdb_id", DataType::Int32, true),
@@ -93,11 +101,11 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"comp_cast_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("kind", DataType::Utf8, false),
]),
"company_name" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("name", DataType::Utf8, false),
Field::new("country_code", DataType::Utf8, true),
Field::new("imdb_id", DataType::Int32, true),
@@ -106,59 +114,59 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"company_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("kind", DataType::Utf8, true),
]),
"complete_cast" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, true),
Field::new("subject_id", DataType::Int32, false),
Field::new("status_id", DataType::Int32, false),
]),
"info_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("info", DataType::Utf8, false),
]),
"keyword" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("keyword", DataType::Utf8, false),
Field::new("phonetic_code", DataType::Utf8, true),
]),
"kind_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("kind", DataType::Utf8, true),
]),
"link_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("link", DataType::Utf8, false),
]),
"movie_companies" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("company_id", DataType::Int32, false),
Field::new("company_type_id", DataType::Int32, false),
Field::new("note", DataType::Utf8, true),
]),
"movie_info_idx" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("info_type_id", DataType::Int32, false),
Field::new("info", DataType::Utf8, false),
Field::new("note", DataType::Utf8, true),
]),
"movie_keyword" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("keyword_id", DataType::Int32, false),
]),
"movie_link" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("linked_movie_id", DataType::Int32, false),
Field::new("link_type_id", DataType::Int32, false),
]),
"name" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("name", DataType::Utf8, false),
Field::new("imdb_index", DataType::Utf8, true),
Field::new("imdb_id", DataType::Int32, true),
@@ -169,11 +177,11 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"role_type" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("role", DataType::Utf8, false),
]),
"title" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("title", DataType::Utf8, false),
Field::new("imdb_index", DataType::Utf8, true),
Field::new("kind_id", DataType::Int32, false),
@@ -187,14 +195,14 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
Field::new("md5sum", DataType::Utf8, true),
]),
"movie_info" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("movie_id", DataType::Int32, false),
Field::new("info_type_id", DataType::Int32, false),
Field::new("info", DataType::Utf8, false),
Field::new("note", DataType::Utf8, true),
]),
"person_info" => Schema::new(vec![
- Field::new("id", DataType::Int32, false),
+ Field::new("id", DataType::UInt32, false),
Field::new("person_id", DataType::Int32, false),
Field::new("info_type_id", DataType::Int32, false),
Field::new("info", DataType::Utf8, false),
@@ -203,3 +211,26 @@ pub fn get_imdb_table_schema(table: &str) -> Schema {
_ => unimplemented!("Schema for table {} is not implemented", table),
}
}
+
+/// Get the SQL statements from the specified query file
+pub fn get_query_sql(query: &str) -> Result<Vec<String>> {
+ let possibilities = vec![
+ format!("queries/imdb/{query}.sql"),
+ format!("benchmarks/queries/imdb/{query}.sql"),
+ ];
+ let mut errors = vec![];
+ for filename in possibilities {
+ match fs::read_to_string(&filename) {
+ Ok(contents) => {
+ return Ok(contents
+ .split(';')
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+ .map(|s| s.to_string())
+ .collect());
+ }
+ Err(e) => errors.push(format!("{filename}: {e}")),
+ };
+ }
+ plan_err!("invalid query. Could not find query: {:?}", errors)
+}
diff --git a/benchmarks/src/imdb/run.rs b/benchmarks/src/imdb/run.rs
new file mode 100644
index 0000000000..697c79dc89
--- /dev/null
+++ b/benchmarks/src/imdb/run.rs
@@ -0,0 +1,827 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::path::PathBuf;
+use std::sync::Arc;
+
+use super::{get_imdb_table_schema, get_query_sql, IMDB_TABLES};
+use crate::{BenchmarkRun, CommonOpt};
+
+use arrow::record_batch::RecordBatch;
+use arrow::util::pretty::{self, pretty_format_batches};
+use datafusion::datasource::file_format::csv::CsvFormat;
+use datafusion::datasource::file_format::parquet::ParquetFormat;
+use datafusion::datasource::file_format::FileFormat;
+use datafusion::datasource::listing::{
+ ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
+};
+use datafusion::datasource::{MemTable, TableProvider};
+use datafusion::error::Result;
+use datafusion::physical_plan::display::DisplayableExecutionPlan;
+use datafusion::physical_plan::{collect, displayable};
+use datafusion::prelude::*;
+use datafusion_common::instant::Instant;
+use datafusion_common::{DEFAULT_CSV_EXTENSION, DEFAULT_PARQUET_EXTENSION};
+
+use log::info;
+use structopt::StructOpt;
+
+// hack to avoid `default_value is meaningless for bool` errors
+type BoolDefaultTrue = bool;
+
+/// Run the imdb benchmark (a.k.a. JOB).
+///
+/// This benchmarks is derived from the [Join Order Benchmark / JOB] proposed in paper [How Good Are Query Optimizers, Really?][1].
+/// The data and answers are downloaded from
+/// [2] and [3].
+///
+/// [1]: https://www.vldb.org/pvldb/vol9/p204-leis.pdf
+/// [2]: http://homepages.cwi.nl/~boncz/job/imdb.tgz
+/// [3]: https://db.in.tum.de/~leis/qo/job.tgz
+
+#[derive(Debug, StructOpt, Clone)]
+#[structopt(verbatim_doc_comment)]
+pub struct RunOpt {
+ /// Query number. If not specified, runs all queries
+ #[structopt(short, long)]
+ query: Option<usize>,
+
+ /// Common options
+ #[structopt(flatten)]
+ common: CommonOpt,
+
+ /// Path to data files
+ #[structopt(parse(from_os_str), required = true, short = "p", long = "path")]
+ path: PathBuf,
+
+ /// File format: `csv` or `parquet`
+ #[structopt(short = "f", long = "format", default_value = "csv")]
+ file_format: String,
+
+ /// Load the data into a MemTable before executing the query
+ #[structopt(short = "m", long = "mem-table")]
+ mem_table: bool,
+
+ /// Path to machine readable output file
+ #[structopt(parse(from_os_str), short = "o", long = "output")]
+ output_path: Option<PathBuf>,
+
+ /// Whether to disable collection of statistics (and cost based optimizations) or not.
+ #[structopt(short = "S", long = "disable-statistics")]
+ disable_statistics: bool,
+
+ /// If true then hash join used, if false then sort merge join
+ /// True by default.
+ #[structopt(short = "j", long = "prefer_hash_join", default_value = "true")]
+ prefer_hash_join: BoolDefaultTrue,
+}
+
+const IMDB_QUERY_START_ID: usize = 1;
+const IMDB_QUERY_END_ID: usize = 113;
+
+fn map_query_id_to_str(query_id: usize) -> &'static str {
+ match query_id {
+ // 1
+ 1 => "1a",
+ 2 => "1b",
+ 3 => "1c",
+ 4 => "1d",
+
+ // 2
+ 5 => "2a",
+ 6 => "2b",
+ 7 => "2c",
+ 8 => "2d",
+
+ // 3
+ 9 => "3a",
+ 10 => "3b",
+ 11 => "3c",
+
+ // 4
+ 12 => "4a",
+ 13 => "4b",
+ 14 => "4c",
+
+ // 5
+ 15 => "5a",
+ 16 => "5b",
+ 17 => "5c",
+
+ // 6
+ 18 => "6a",
+ 19 => "6b",
+ 20 => "6c",
+ 21 => "6d",
+ 22 => "6e",
+ 23 => "6f",
+
+ // 7
+ 24 => "7a",
+ 25 => "7b",
+ 26 => "7c",
+
+ // 8
+ 27 => "8a",
+ 28 => "8b",
+ 29 => "8c",
+ 30 => "8d",
+
+ // 9
+ 31 => "9a",
+ 32 => "9b",
+ 33 => "9c",
+ 34 => "9d",
+
+ // 10
+ 35 => "10a",
+ 36 => "10b",
+ 37 => "10c",
+
+ // 11
+ 38 => "11a",
+ 39 => "11b",
+ 40 => "11c",
+ 41 => "11d",
+
+ // 12
+ 42 => "12a",
+ 43 => "12b",
+ 44 => "12c",
+
+ // 13
+ 45 => "13a",
+ 46 => "13b",
+ 47 => "13c",
+ 48 => "13d",
+
+ // 14
+ 49 => "14a",
+ 50 => "14b",
+ 51 => "14c",
+
+ // 15
+ 52 => "15a",
+ 53 => "15b",
+ 54 => "15c",
+ 55 => "15d",
+
+ // 16
+ 56 => "16a",
+ 57 => "16b",
+ 58 => "16c",
+ 59 => "16d",
+
+ // 17
+ 60 => "17a",
+ 61 => "17b",
+ 62 => "17c",
+ 63 => "17d",
+ 64 => "17e",
+ 65 => "17f",
+
+ // 18
+ 66 => "18a",
+ 67 => "18b",
+ 68 => "18c",
+
+ // 19
+ 69 => "19a",
+ 70 => "19b",
+ 71 => "19c",
+ 72 => "19d",
+
+ // 20
+ 73 => "20a",
+ 74 => "20b",
+ 75 => "20c",
+
+ // 21
+ 76 => "21a",
+ 77 => "21b",
+ 78 => "21c",
+
+ // 22
+ 79 => "22a",
+ 80 => "22b",
+ 81 => "22c",
+ 82 => "22d",
+
+ // 23
+ 83 => "23a",
+ 84 => "23b",
+ 85 => "23c",
+
+ // 24
+ 86 => "24a",
+ 87 => "24b",
+
+ // 25
+ 88 => "25a",
+ 89 => "25b",
+ 90 => "25c",
+
+ // 26
+ 91 => "26a",
+ 92 => "26b",
+ 93 => "26c",
+
+ // 27
+ 94 => "27a",
+ 95 => "27b",
+ 96 => "27c",
+
+ // 28
+ 97 => "28a",
+ 98 => "28b",
+ 99 => "28c",
+
+ // 29
+ 100 => "29a",
+ 101 => "29b",
+ 102 => "29c",
+
+ // 30
+ 103 => "30a",
+ 104 => "30b",
+ 105 => "30c",
+
+ // 31
+ 106 => "31a",
+ 107 => "31b",
+ 108 => "31c",
+
+ // 32
+ 109 => "32a",
+ 110 => "32b",
+
+ // 33
+ 111 => "33a",
+ 112 => "33b",
+ 113 => "33c",
+
+ // Fallback for unknown query_id
+ _ => "unknown",
+ }
+}
+
+impl RunOpt {
+ pub async fn run(self) -> Result<()> {
+ println!("Running benchmarks with the following options: {self:?}");
+ let query_range = match self.query {
+ Some(query_id) => query_id..=query_id,
+ None => IMDB_QUERY_START_ID..=IMDB_QUERY_END_ID,
+ };
+
+ let mut benchmark_run = BenchmarkRun::new();
+ for query_id in query_range {
+ benchmark_run.start_new_case(&format!("Query {query_id}"));
+ let query_run = self.benchmark_query(query_id).await?;
+ for iter in query_run {
+ benchmark_run.write_iter(iter.elapsed, iter.row_count);
+ }
+ }
+ benchmark_run.maybe_write_json(self.output_path.as_ref())?;
+ Ok(())
+ }
+
+ async fn benchmark_query(&self, query_id: usize) -> Result<Vec<QueryResult>> {
+ let mut config = self
+ .common
+ .config()
+ .with_collect_statistics(!self.disable_statistics);
+ config.options_mut().optimizer.prefer_hash_join = self.prefer_hash_join;
+ config
+ .options_mut()
+ .execution
+ .parquet
+ .schema_force_view_types = self.common.force_view_types;
+ let ctx = SessionContext::new_with_config(config);
+
+ // register tables
+ self.register_tables(&ctx).await?;
+
+ let mut millis = vec![];
+ // run benchmark
+ let mut query_results = vec![];
+ for i in 0..self.iterations() {
+ let start = Instant::now();
+
+ let query_id_str = map_query_id_to_str(query_id);
+ let sql = &get_query_sql(query_id_str)?;
+
+ let mut result = vec![];
+
+ for query in sql {
+ result = self.execute_query(&ctx, query).await?;
+ }
+
+ let elapsed = start.elapsed(); //.as_secs_f64() * 1000.0;
+ let ms = elapsed.as_secs_f64() * 1000.0;
+ millis.push(ms);
+ info!("output:\n\n{}\n\n", pretty_format_batches(&result)?);
+ let row_count = result.iter().map(|b| b.num_rows()).sum();
+ println!(
+ "Query {query_id} iteration {i} took {ms:.1} ms and returned {row_count} rows"
+ );
+ query_results.push(QueryResult { elapsed, row_count });
+ }
+
+ let avg = millis.iter().sum::<f64>() / millis.len() as f64;
+ println!("Query {query_id} avg time: {avg:.2} ms");
+
+ Ok(query_results)
+ }
+
+ async fn register_tables(&self, ctx: &SessionContext) -> Result<()> {
+ for table in IMDB_TABLES {
+ let table_provider = { self.get_table(ctx, table).await? };
+
+ if self.mem_table {
+ println!("Loading table '{table}' into memory");
+ let start = Instant::now();
+ let memtable =
+ MemTable::load(table_provider, Some(self.partitions()), &ctx.state())
+ .await?;
+ println!(
+ "Loaded table '{}' into memory in {} ms",
+ table,
+ start.elapsed().as_millis()
+ );
+ ctx.register_table(*table, Arc::new(memtable))?;
+ } else {
+ ctx.register_table(*table, table_provider)?;
+ }
+ }
+ Ok(())
+ }
+
+ async fn execute_query(
+ &self,
+ ctx: &SessionContext,
+ sql: &str,
+ ) -> Result<Vec<RecordBatch>> {
+ let debug = self.common.debug;
+ let plan = ctx.sql(sql).await?;
+ let (state, plan) = plan.into_parts();
+
+ if debug {
+ println!("=== Logical plan ===\n{plan}\n");
+ }
+
+ let plan = state.optimize(&plan)?;
+ if debug {
+ println!("=== Optimized logical plan ===\n{plan}\n");
+ }
+ let physical_plan = state.create_physical_plan(&plan).await?;
+ if debug {
+ println!(
+ "=== Physical plan ===\n{}\n",
+ displayable(physical_plan.as_ref()).indent(true)
+ );
+ }
+ let result = collect(physical_plan.clone(), state.task_ctx()).await?;
+ if debug {
+ println!(
+ "=== Physical plan with metrics ===\n{}\n",
+ DisplayableExecutionPlan::with_metrics(physical_plan.as_ref())
+ .indent(true)
+ );
+ if !result.is_empty() {
+ // do not call print_batches if there are no batches as the result is confusing
+ // and makes it look like there is a batch with no columns
+ pretty::print_batches(&result)?;
+ }
+ }
+ Ok(result)
+ }
+
+ async fn get_table(
+ &self,
+ ctx: &SessionContext,
+ table: &str,
+ ) -> Result<Arc<dyn TableProvider>> {
+ let path = self.path.to_str().unwrap();
+ let table_format = self.file_format.as_str();
+
+ // Obtain a snapshot of the SessionState
+ let state = ctx.state();
+ let (format, path, extension): (Arc<dyn FileFormat>, String, &'static str) =
+ match table_format {
+ // dbgen creates .tbl ('|' delimited) files without header
+ "tbl" => {
+ let path = format!("{path}/{table}.tbl");
+
+ let format = CsvFormat::default()
+ .with_delimiter(b'|')
+ .with_has_header(false);
+
+ (Arc::new(format), path, ".tbl")
+ }
+ "csv" => {
+ let path = format!("{path}/{table}.csv");
+ let format = CsvFormat::default()
+ .with_delimiter(b',')
+ .with_escape(Some(b'\\'))
+ .with_has_header(false);
+
+ (Arc::new(format), path, DEFAULT_CSV_EXTENSION)
+ }
+ "parquet" => {
+ let path = format!("{path}/{table}.parquet");
+ let format = ParquetFormat::default()
+ .with_options(ctx.state().table_options().parquet.clone());
+ (Arc::new(format), path, DEFAULT_PARQUET_EXTENSION)
+ }
+ other => {
+ unimplemented!("Invalid file format '{}'", other);
+ }
+ };
+
+ let options = ListingOptions::new(format)
+ .with_file_extension(extension)
+ .with_collect_stat(state.config().collect_statistics());
+
+ let table_path = ListingTableUrl::parse(path)?;
+ let config = ListingTableConfig::new(table_path).with_listing_options(options);
+ let config = match table_format {
+ "parquet" => config.with_schema(Arc::new(get_imdb_table_schema(table))),
+ "csv" => config.with_schema(Arc::new(get_imdb_table_schema(table))),
+ _ => unreachable!(),
+ };
+
+ Ok(Arc::new(ListingTable::try_new(config)?))
+ }
+
+ fn iterations(&self) -> usize {
+ self.common.iterations
+ }
+
+ fn partitions(&self) -> usize {
+ self.common.partitions.unwrap_or(num_cpus::get())
+ }
+}
+
+struct QueryResult {
+ elapsed: std::time::Duration,
+ row_count: usize,
+}
+
+#[cfg(test)]
+// Only run with "ci" mode when we have the data
+#[cfg(feature = "ci")]
+mod tests {
+ use std::path::Path;
+
+ use super::*;
+
+ use datafusion::common::exec_err;
+ use datafusion::error::Result;
+ use datafusion_proto::bytes::{
+ logical_plan_from_bytes, logical_plan_to_bytes, physical_plan_from_bytes,
+ physical_plan_to_bytes,
+ };
+
+ fn get_imdb_data_path() -> Result<String> {
+ let path =
+ std::env::var("IMDB_DATA").unwrap_or_else(|_| "benchmarks/data".to_string());
+ if !Path::new(&path).exists() {
+ return exec_err!(
+ "Benchmark data not found (set IMDB_DATA env var to override): {}",
+ path
+ );
+ }
+ Ok(path)
+ }
+
+ async fn round_trip_logical_plan(query: usize) -> Result<()> {
+ let ctx = SessionContext::default();
+ let path = get_imdb_data_path()?;
+ let common = CommonOpt {
+ iterations: 1,
+ partitions: Some(2),
+ batch_size: 8192,
+ debug: false,
+ force_view_types: false,
+ };
+ let opt = RunOpt {
+ query: Some(query),
+ common,
+ path: PathBuf::from(path.to_string()),
+ file_format: "parquet".to_string(),
+ mem_table: false,
+ output_path: None,
+ disable_statistics: false,
+ prefer_hash_join: true,
+ };
+ opt.register_tables(&ctx).await?;
+ let queries = get_query_sql(map_query_id_to_str(query))?;
+ for query in queries {
+ let plan = ctx.sql(&query).await?;
+ let plan = plan.into_optimized_plan()?;
+ let bytes = logical_plan_to_bytes(&plan)?;
+ let plan2 = logical_plan_from_bytes(&bytes, &ctx)?;
+ let plan_formatted = format!("{}", plan.display_indent());
+ let plan2_formatted = format!("{}", plan2.display_indent());
+ assert_eq!(plan_formatted, plan2_formatted);
+ }
+ Ok(())
+ }
+
+ async fn round_trip_physical_plan(query: usize) -> Result<()> {
+ let ctx = SessionContext::default();
+ let path = get_imdb_data_path()?;
+ let common = CommonOpt {
+ iterations: 1,
+ partitions: Some(2),
+ batch_size: 8192,
+ debug: false,
+ force_view_types: false,
+ };
+ let opt = RunOpt {
+ query: Some(query),
+ common,
+ path: PathBuf::from(path.to_string()),
+ file_format: "parquet".to_string(),
+ mem_table: false,
+ output_path: None,
+ disable_statistics: false,
+ prefer_hash_join: true,
+ };
+ opt.register_tables(&ctx).await?;
+ let queries = get_query_sql(map_query_id_to_str(query))?;
+ for query in queries {
+ let plan = ctx.sql(&query).await?;
+ let plan = plan.create_physical_plan().await?;
+ let bytes = physical_plan_to_bytes(plan.clone())?;
+ let plan2 = physical_plan_from_bytes(&bytes, &ctx)?;
+ let plan_formatted = format!("{}", displayable(plan.as_ref()).indent(false));
+ let plan2_formatted =
+ format!("{}", displayable(plan2.as_ref()).indent(false));
+ assert_eq!(plan_formatted, plan2_formatted);
+ }
+ Ok(())
+ }
+
+ macro_rules! test_round_trip_logical {
+ ($tn:ident, $query:expr) => {
+ #[tokio::test]
+ async fn $tn() -> Result<()> {
+ round_trip_logical_plan($query).await
+ }
+ };
+ }
+
+ macro_rules! test_round_trip_physical {
+ ($tn:ident, $query:expr) => {
+ #[tokio::test]
+ async fn $tn() -> Result<()> {
+ round_trip_physical_plan($query).await
+ }
+ };
+ }
+
+ // logical plan tests
+ test_round_trip_logical!(round_trip_logical_plan_1a, 1);
+ test_round_trip_logical!(round_trip_logical_plan_1b, 2);
+ test_round_trip_logical!(round_trip_logical_plan_1c, 3);
+ test_round_trip_logical!(round_trip_logical_plan_1d, 4);
+ test_round_trip_logical!(round_trip_logical_plan_2a, 5);
+ test_round_trip_logical!(round_trip_logical_plan_2b, 6);
+ test_round_trip_logical!(round_trip_logical_plan_2c, 7);
+ test_round_trip_logical!(round_trip_logical_plan_2d, 8);
+ test_round_trip_logical!(round_trip_logical_plan_3a, 9);
+ test_round_trip_logical!(round_trip_logical_plan_3b, 10);
+ test_round_trip_logical!(round_trip_logical_plan_3c, 11);
+ test_round_trip_logical!(round_trip_logical_plan_4a, 12);
+ test_round_trip_logical!(round_trip_logical_plan_4b, 13);
+ test_round_trip_logical!(round_trip_logical_plan_4c, 14);
+ test_round_trip_logical!(round_trip_logical_plan_5a, 15);
+ test_round_trip_logical!(round_trip_logical_plan_5b, 16);
+ test_round_trip_logical!(round_trip_logical_plan_5c, 17);
+ test_round_trip_logical!(round_trip_logical_plan_6a, 18);
+ test_round_trip_logical!(round_trip_logical_plan_6b, 19);
+ test_round_trip_logical!(round_trip_logical_plan_6c, 20);
+ test_round_trip_logical!(round_trip_logical_plan_6d, 21);
+ test_round_trip_logical!(round_trip_logical_plan_6e, 22);
+ test_round_trip_logical!(round_trip_logical_plan_6f, 23);
+ test_round_trip_logical!(round_trip_logical_plan_7a, 24);
+ test_round_trip_logical!(round_trip_logical_plan_7b, 25);
+ test_round_trip_logical!(round_trip_logical_plan_7c, 26);
+ test_round_trip_logical!(round_trip_logical_plan_8a, 27);
+ test_round_trip_logical!(round_trip_logical_plan_8b, 28);
+ test_round_trip_logical!(round_trip_logical_plan_8c, 29);
+ test_round_trip_logical!(round_trip_logical_plan_8d, 30);
+ test_round_trip_logical!(round_trip_logical_plan_9a, 31);
+ test_round_trip_logical!(round_trip_logical_plan_9b, 32);
+ test_round_trip_logical!(round_trip_logical_plan_9c, 33);
+ test_round_trip_logical!(round_trip_logical_plan_9d, 34);
+ test_round_trip_logical!(round_trip_logical_plan_10a, 35);
+ test_round_trip_logical!(round_trip_logical_plan_10b, 36);
+ test_round_trip_logical!(round_trip_logical_plan_10c, 37);
+ test_round_trip_logical!(round_trip_logical_plan_11a, 38);
+ test_round_trip_logical!(round_trip_logical_plan_11b, 39);
+ test_round_trip_logical!(round_trip_logical_plan_11c, 40);
+ test_round_trip_logical!(round_trip_logical_plan_11d, 41);
+ test_round_trip_logical!(round_trip_logical_plan_12a, 42);
+ test_round_trip_logical!(round_trip_logical_plan_12b, 43);
+ test_round_trip_logical!(round_trip_logical_plan_12c, 44);
+ test_round_trip_logical!(round_trip_logical_plan_13a, 45);
+ test_round_trip_logical!(round_trip_logical_plan_13b, 46);
+ test_round_trip_logical!(round_trip_logical_plan_13c, 47);
+ test_round_trip_logical!(round_trip_logical_plan_13d, 48);
+ test_round_trip_logical!(round_trip_logical_plan_14a, 49);
+ test_round_trip_logical!(round_trip_logical_plan_14b, 50);
+ test_round_trip_logical!(round_trip_logical_plan_14c, 51);
+ test_round_trip_logical!(round_trip_logical_plan_15a, 52);
+ test_round_trip_logical!(round_trip_logical_plan_15b, 53);
+ test_round_trip_logical!(round_trip_logical_plan_15c, 54);
+ test_round_trip_logical!(round_trip_logical_plan_15d, 55);
+ test_round_trip_logical!(round_trip_logical_plan_16a, 56);
+ test_round_trip_logical!(round_trip_logical_plan_16b, 57);
+ test_round_trip_logical!(round_trip_logical_plan_16c, 58);
+ test_round_trip_logical!(round_trip_logical_plan_16d, 59);
+ test_round_trip_logical!(round_trip_logical_plan_17a, 60);
+ test_round_trip_logical!(round_trip_logical_plan_17b, 61);
+ test_round_trip_logical!(round_trip_logical_plan_17c, 62);
+ test_round_trip_logical!(round_trip_logical_plan_17d, 63);
+ test_round_trip_logical!(round_trip_logical_plan_17e, 64);
+ test_round_trip_logical!(round_trip_logical_plan_17f, 65);
+ test_round_trip_logical!(round_trip_logical_plan_18a, 66);
+ test_round_trip_logical!(round_trip_logical_plan_18b, 67);
+ test_round_trip_logical!(round_trip_logical_plan_18c, 68);
+ test_round_trip_logical!(round_trip_logical_plan_19a, 69);
+ test_round_trip_logical!(round_trip_logical_plan_19b, 70);
+ test_round_trip_logical!(round_trip_logical_plan_19c, 71);
+ test_round_trip_logical!(round_trip_logical_plan_19d, 72);
+ test_round_trip_logical!(round_trip_logical_plan_20a, 73);
+ test_round_trip_logical!(round_trip_logical_plan_20b, 74);
+ test_round_trip_logical!(round_trip_logical_plan_20c, 75);
+ test_round_trip_logical!(round_trip_logical_plan_21a, 76);
+ test_round_trip_logical!(round_trip_logical_plan_21b, 77);
+ test_round_trip_logical!(round_trip_logical_plan_21c, 78);
+ test_round_trip_logical!(round_trip_logical_plan_22a, 79);
+ test_round_trip_logical!(round_trip_logical_plan_22b, 80);
+ test_round_trip_logical!(round_trip_logical_plan_22c, 81);
+ test_round_trip_logical!(round_trip_logical_plan_22d, 82);
+ test_round_trip_logical!(round_trip_logical_plan_23a, 83);
+ test_round_trip_logical!(round_trip_logical_plan_23b, 84);
+ test_round_trip_logical!(round_trip_logical_plan_23c, 85);
+ test_round_trip_logical!(round_trip_logical_plan_24a, 86);
+ test_round_trip_logical!(round_trip_logical_plan_24b, 87);
+ test_round_trip_logical!(round_trip_logical_plan_25a, 88);
+ test_round_trip_logical!(round_trip_logical_plan_25b, 89);
+ test_round_trip_logical!(round_trip_logical_plan_25c, 90);
+ test_round_trip_logical!(round_trip_logical_plan_26a, 91);
+ test_round_trip_logical!(round_trip_logical_plan_26b, 92);
+ test_round_trip_logical!(round_trip_logical_plan_26c, 93);
+ test_round_trip_logical!(round_trip_logical_plan_27a, 94);
+ test_round_trip_logical!(round_trip_logical_plan_27b, 95);
+ test_round_trip_logical!(round_trip_logical_plan_27c, 96);
+ test_round_trip_logical!(round_trip_logical_plan_28a, 97);
+ test_round_trip_logical!(round_trip_logical_plan_28b, 98);
+ test_round_trip_logical!(round_trip_logical_plan_28c, 99);
+ test_round_trip_logical!(round_trip_logical_plan_29a, 100);
+ test_round_trip_logical!(round_trip_logical_plan_29b, 101);
+ test_round_trip_logical!(round_trip_logical_plan_29c, 102);
+ test_round_trip_logical!(round_trip_logical_plan_30a, 103);
+ test_round_trip_logical!(round_trip_logical_plan_30b, 104);
+ test_round_trip_logical!(round_trip_logical_plan_30c, 105);
+ test_round_trip_logical!(round_trip_logical_plan_31a, 106);
+ test_round_trip_logical!(round_trip_logical_plan_31b, 107);
+ test_round_trip_logical!(round_trip_logical_plan_31c, 108);
+ test_round_trip_logical!(round_trip_logical_plan_32a, 109);
+ test_round_trip_logical!(round_trip_logical_plan_32b, 110);
+ test_round_trip_logical!(round_trip_logical_plan_33a, 111);
+ test_round_trip_logical!(round_trip_logical_plan_33b, 112);
+ test_round_trip_logical!(round_trip_logical_plan_33c, 113);
+
+ // physical plan tests
+ test_round_trip_physical!(round_trip_physical_plan_1a, 1);
+ test_round_trip_physical!(round_trip_physical_plan_1b, 2);
+ test_round_trip_physical!(round_trip_physical_plan_1c, 3);
+ test_round_trip_physical!(round_trip_physical_plan_1d, 4);
+ test_round_trip_physical!(round_trip_physical_plan_2a, 5);
+ test_round_trip_physical!(round_trip_physical_plan_2b, 6);
+ test_round_trip_physical!(round_trip_physical_plan_2c, 7);
+ test_round_trip_physical!(round_trip_physical_plan_2d, 8);
+ test_round_trip_physical!(round_trip_physical_plan_3a, 9);
+ test_round_trip_physical!(round_trip_physical_plan_3b, 10);
+ test_round_trip_physical!(round_trip_physical_plan_3c, 11);
+ test_round_trip_physical!(round_trip_physical_plan_4a, 12);
+ test_round_trip_physical!(round_trip_physical_plan_4b, 13);
+ test_round_trip_physical!(round_trip_physical_plan_4c, 14);
+ test_round_trip_physical!(round_trip_physical_plan_5a, 15);
+ test_round_trip_physical!(round_trip_physical_plan_5b, 16);
+ test_round_trip_physical!(round_trip_physical_plan_5c, 17);
+ test_round_trip_physical!(round_trip_physical_plan_6a, 18);
+ test_round_trip_physical!(round_trip_physical_plan_6b, 19);
+ test_round_trip_physical!(round_trip_physical_plan_6c, 20);
+ test_round_trip_physical!(round_trip_physical_plan_6d, 21);
+ test_round_trip_physical!(round_trip_physical_plan_6e, 22);
+ test_round_trip_physical!(round_trip_physical_plan_6f, 23);
+ test_round_trip_physical!(round_trip_physical_plan_7a, 24);
+ test_round_trip_physical!(round_trip_physical_plan_7b, 25);
+ test_round_trip_physical!(round_trip_physical_plan_7c, 26);
+ test_round_trip_physical!(round_trip_physical_plan_8a, 27);
+ test_round_trip_physical!(round_trip_physical_plan_8b, 28);
+ test_round_trip_physical!(round_trip_physical_plan_8c, 29);
+ test_round_trip_physical!(round_trip_physical_plan_8d, 30);
+ test_round_trip_physical!(round_trip_physical_plan_9a, 31);
+ test_round_trip_physical!(round_trip_physical_plan_9b, 32);
+ test_round_trip_physical!(round_trip_physical_plan_9c, 33);
+ test_round_trip_physical!(round_trip_physical_plan_9d, 34);
+ test_round_trip_physical!(round_trip_physical_plan_10a, 35);
+ test_round_trip_physical!(round_trip_physical_plan_10b, 36);
+ test_round_trip_physical!(round_trip_physical_plan_10c, 37);
+ test_round_trip_physical!(round_trip_physical_plan_11a, 38);
+ test_round_trip_physical!(round_trip_physical_plan_11b, 39);
+ test_round_trip_physical!(round_trip_physical_plan_11c, 40);
+ test_round_trip_physical!(round_trip_physical_plan_11d, 41);
+ test_round_trip_physical!(round_trip_physical_plan_12a, 42);
+ test_round_trip_physical!(round_trip_physical_plan_12b, 43);
+ test_round_trip_physical!(round_trip_physical_plan_12c, 44);
+ test_round_trip_physical!(round_trip_physical_plan_13a, 45);
+ test_round_trip_physical!(round_trip_physical_plan_13b, 46);
+ test_round_trip_physical!(round_trip_physical_plan_13c, 47);
+ test_round_trip_physical!(round_trip_physical_plan_13d, 48);
+ test_round_trip_physical!(round_trip_physical_plan_14a, 49);
+ test_round_trip_physical!(round_trip_physical_plan_14b, 50);
+ test_round_trip_physical!(round_trip_physical_plan_14c, 51);
+ test_round_trip_physical!(round_trip_physical_plan_15a, 52);
+ test_round_trip_physical!(round_trip_physical_plan_15b, 53);
+ test_round_trip_physical!(round_trip_physical_plan_15c, 54);
+ test_round_trip_physical!(round_trip_physical_plan_15d, 55);
+ test_round_trip_physical!(round_trip_physical_plan_16a, 56);
+ test_round_trip_physical!(round_trip_physical_plan_16b, 57);
+ test_round_trip_physical!(round_trip_physical_plan_16c, 58);
+ test_round_trip_physical!(round_trip_physical_plan_16d, 59);
+ test_round_trip_physical!(round_trip_physical_plan_17a, 60);
+ test_round_trip_physical!(round_trip_physical_plan_17b, 61);
+ test_round_trip_physical!(round_trip_physical_plan_17c, 62);
+ test_round_trip_physical!(round_trip_physical_plan_17d, 63);
+ test_round_trip_physical!(round_trip_physical_plan_17e, 64);
+ test_round_trip_physical!(round_trip_physical_plan_17f, 65);
+ test_round_trip_physical!(round_trip_physical_plan_18a, 66);
+ test_round_trip_physical!(round_trip_physical_plan_18b, 67);
+ test_round_trip_physical!(round_trip_physical_plan_18c, 68);
+ test_round_trip_physical!(round_trip_physical_plan_19a, 69);
+ test_round_trip_physical!(round_trip_physical_plan_19b, 70);
+ test_round_trip_physical!(round_trip_physical_plan_19c, 71);
+ test_round_trip_physical!(round_trip_physical_plan_19d, 72);
+ test_round_trip_physical!(round_trip_physical_plan_20a, 73);
+ test_round_trip_physical!(round_trip_physical_plan_20b, 74);
+ test_round_trip_physical!(round_trip_physical_plan_20c, 75);
+ test_round_trip_physical!(round_trip_physical_plan_21a, 76);
+ test_round_trip_physical!(round_trip_physical_plan_21b, 77);
+ test_round_trip_physical!(round_trip_physical_plan_21c, 78);
+ test_round_trip_physical!(round_trip_physical_plan_22a, 79);
+ test_round_trip_physical!(round_trip_physical_plan_22b, 80);
+ test_round_trip_physical!(round_trip_physical_plan_22c, 81);
+ test_round_trip_physical!(round_trip_physical_plan_22d, 82);
+ test_round_trip_physical!(round_trip_physical_plan_23a, 83);
+ test_round_trip_physical!(round_trip_physical_plan_23b, 84);
+ test_round_trip_physical!(round_trip_physical_plan_23c, 85);
+ test_round_trip_physical!(round_trip_physical_plan_24a, 86);
+ test_round_trip_physical!(round_trip_physical_plan_24b, 87);
+ test_round_trip_physical!(round_trip_physical_plan_25a, 88);
+ test_round_trip_physical!(round_trip_physical_plan_25b, 89);
+ test_round_trip_physical!(round_trip_physical_plan_25c, 90);
+ test_round_trip_physical!(round_trip_physical_plan_26a, 91);
+ test_round_trip_physical!(round_trip_physical_plan_26b, 92);
+ test_round_trip_physical!(round_trip_physical_plan_26c, 93);
+ test_round_trip_physical!(round_trip_physical_plan_27a, 94);
+ test_round_trip_physical!(round_trip_physical_plan_27b, 95);
+ test_round_trip_physical!(round_trip_physical_plan_27c, 96);
+ test_round_trip_physical!(round_trip_physical_plan_28a, 97);
+ test_round_trip_physical!(round_trip_physical_plan_28b, 98);
+ test_round_trip_physical!(round_trip_physical_plan_28c, 99);
+ test_round_trip_physical!(round_trip_physical_plan_29a, 100);
+ test_round_trip_physical!(round_trip_physical_plan_29b, 101);
+ test_round_trip_physical!(round_trip_physical_plan_29c, 102);
+ test_round_trip_physical!(round_trip_physical_plan_30a, 103);
+ test_round_trip_physical!(round_trip_physical_plan_30b, 104);
+ test_round_trip_physical!(round_trip_physical_plan_30c, 105);
+ test_round_trip_physical!(round_trip_physical_plan_31a, 106);
+ test_round_trip_physical!(round_trip_physical_plan_31b, 107);
+ test_round_trip_physical!(round_trip_physical_plan_31c, 108);
+ test_round_trip_physical!(round_trip_physical_plan_32a, 109);
+ test_round_trip_physical!(round_trip_physical_plan_32b, 110);
+ test_round_trip_physical!(round_trip_physical_plan_33a, 111);
+ test_round_trip_physical!(round_trip_physical_plan_33b, 112);
+ test_round_trip_physical!(round_trip_physical_plan_33c, 113);
+}
diff --git a/datafusion-cli/Cargo.lock b/datafusion-cli/Cargo.lock
index fbe7d5c04b..8a6ccacbb3 100644
--- a/datafusion-cli/Cargo.lock
+++ b/datafusion-cli/Cargo.lock
@@ -4,9 +4,9 @@ version = 3
[[package]]
name = "addr2line"
-version = "0.24.1"
+version = "0.24.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f5fb1d8e4442bd405fdfd1dacb42792696b0cf9cb15882e5d097b742a676d375"
+checksum = "dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1"
dependencies = [
"gimli",
]
@@ -173,9 +173,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "arrow"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "45aef0d9cf9a039bf6cd1acc451b137aca819977b0928dece52bd92811b640ba"
+checksum = "a9ba0d7248932f4e2a12fb37f0a2e3ec82b3bdedbac2a1dce186e036843b8f8c"
dependencies = [
"arrow-arith",
"arrow-array",
@@ -194,9 +194,9 @@ dependencies = [
[[package]]
name = "arrow-arith"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "03675e42d1560790f3524800e41403b40d0da1c793fe9528929fde06d8c7649a"
+checksum = "d60afcdc004841a5c8d8da4f4fa22d64eb19c0c01ef4bcedd77f175a7cf6e38f"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -209,9 +209,9 @@ dependencies = [
[[package]]
name = "arrow-array"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cd2bf348cf9f02a5975c5962c7fa6dee107a2009a7b41ac5fb1a027e12dc033f"
+checksum = "7f16835e8599dbbb1659fd869d865254c4cf32c6c2bb60b6942ac9fc36bfa5da"
dependencies = [
"ahash",
"arrow-buffer",
@@ -220,15 +220,15 @@ dependencies = [
"chrono",
"chrono-tz",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"num",
]
[[package]]
name = "arrow-buffer"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3092e37715f168976012ce52273c3989b5793b0db5f06cbaa246be25e5f0924d"
+checksum = "1a1f34f0faae77da6b142db61deba2cb6d60167592b178be317b341440acba80"
dependencies = [
"bytes",
"half",
@@ -237,9 +237,9 @@ dependencies = [
[[package]]
name = "arrow-cast"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7ce1018bb710d502f9db06af026ed3561552e493e989a79d0d0f5d9cf267a785"
+checksum = "450e4abb5775bca0740bec0bcf1b1a5ae07eff43bd625661c4436d8e8e4540c4"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -258,9 +258,9 @@ dependencies = [
[[package]]
name = "arrow-csv"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd178575f45624d045e4ebee714e246a05d9652e41363ee3f57ec18cca97f740"
+checksum = "d3a4e4d63830a341713e35d9a42452fbc6241d5f42fa5cf6a4681b8ad91370c4"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -277,9 +277,9 @@ dependencies = [
[[package]]
name = "arrow-data"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4e4ac0c4ee79150afe067dc4857154b3ee9c1cd52b5f40d59a77306d0ed18d65"
+checksum = "2b1e618bbf714c7a9e8d97203c806734f012ff71ae3adc8ad1b075689f540634"
dependencies = [
"arrow-buffer",
"arrow-schema",
@@ -289,9 +289,9 @@ dependencies = [
[[package]]
name = "arrow-ipc"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bb307482348a1267f91b0912e962cd53440e5de0f7fb24c5f7b10da70b38c94a"
+checksum = "f98e983549259a2b97049af7edfb8f28b8911682040e99a94e4ceb1196bd65c2"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -304,9 +304,9 @@ dependencies = [
[[package]]
name = "arrow-json"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d24805ba326758effdd6f2cbdd482fcfab749544f21b134701add25b33f474e6"
+checksum = "b198b9c6fcf086501730efbbcb483317b39330a116125af7bb06467d04b352a3"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -324,9 +324,9 @@ dependencies = [
[[package]]
name = "arrow-ord"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "644046c479d80ae8ed02a7f1e1399072ea344ca6a7b0e293ab2d5d9ed924aa3b"
+checksum = "2427f37b4459a4b9e533045abe87a5183a5e0995a3fc2c2fd45027ae2cc4ef3f"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -339,9 +339,9 @@ dependencies = [
[[package]]
name = "arrow-row"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a29791f8eb13b340ce35525b723f5f0df17ecb955599e11f65c2a94ab34e2efb"
+checksum = "15959657d92e2261a7a323517640af87f5afd9fd8a6492e424ebee2203c567f6"
dependencies = [
"ahash",
"arrow-array",
@@ -353,15 +353,15 @@ dependencies = [
[[package]]
name = "arrow-schema"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c85320a3a2facf2b2822b57aa9d6d9d55edb8aee0b6b5d3b8df158e503d10858"
+checksum = "fbf0388a18fd7f7f3fe3de01852d30f54ed5182f9004db700fbe3ba843ed2794"
[[package]]
name = "arrow-select"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9cc7e6b582e23855fd1625ce46e51647aa440c20ea2e71b1d748e0839dd73cba"
+checksum = "b83e5723d307a38bf00ecd2972cd078d1339c7fd3eb044f609958a9a24463f3a"
dependencies = [
"ahash",
"arrow-array",
@@ -373,9 +373,9 @@ dependencies = [
[[package]]
name = "arrow-string"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0775b6567c66e56ded19b87a954b6b1beffbdd784ef95a3a2b03f59570c1d230"
+checksum = "7ab3db7c09dd826e74079661d84ed01ed06547cf75d52c2818ef776d0d852305"
dependencies = [
"arrow-array",
"arrow-buffer",
@@ -406,9 +406,9 @@ dependencies = [
[[package]]
name = "async-compression"
-version = "0.4.12"
+version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fec134f64e2bc57411226dfc4e52dec859ddfc7e711fc5e07b612584f000e4aa"
+checksum = "7e614738943d3f68c628ae3dbce7c3daffb196665f82f8c8ea6b65de73c79429"
dependencies = [
"bzip2",
"flate2",
@@ -424,9 +424,9 @@ dependencies = [
[[package]]
name = "async-trait"
-version = "0.1.82"
+version = "0.1.83"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a27b8a3a6e1a44fa4c8baf1f653e4172e81486d4941f2237e20dc2d0cf4ddff1"
+checksum = "721cae7de5c34fbb2acd27e21e6d2cf7b886dce0c27388d46c4e6c47ea4318dd"
dependencies = [
"proc-macro2",
"quote",
@@ -450,15 +450,15 @@ checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
[[package]]
name = "autocfg"
-version = "1.3.0"
+version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0c4b4d0bd25bd0b74681c0ad21497610ce1b7c91b1022cd21c80c6fbdd9476b0"
+checksum = "ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26"
[[package]]
name = "aws-config"
-version = "1.5.6"
+version = "1.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "848d7b9b605720989929279fa644ce8f244d0ce3146fcca5b70e4eb7b3c020fc"
+checksum = "7198e6f03240fdceba36656d8be440297b6b82270325908c7381f37d826a74f6"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -523,9 +523,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
-version = "1.43.0"
+version = "1.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "70a9d27ed1c12b1140c47daf1bc541606c43fdafd918c4797d520db0043ceef2"
+checksum = "e33ae899566f3d395cbf42858e433930682cc9c1889fa89318896082fef45efb"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -545,9 +545,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
-version = "1.44.0"
+version = "1.46.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "44514a6ca967686cde1e2a1b81df6ef1883d0e3e570da8d8bc5c491dcb6fc29b"
+checksum = "f39c09e199ebd96b9f860b0fce4b6625f211e064ad7c8693b72ecf7ef03881e0"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -567,9 +567,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
-version = "1.43.0"
+version = "1.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cd7a4d279762a35b9df97209f6808b95d4fe78547fe2316b4d200a0283960c5a"
+checksum = "3d95f93a98130389eb6233b9d615249e543f6c24a68ca1f109af9ca5164a8765"
dependencies = [
"aws-credential-types",
"aws-runtime",
@@ -707,9 +707,9 @@ dependencies = [
[[package]]
name = "aws-smithy-types"
-version = "1.2.6"
+version = "1.2.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "03701449087215b5369c7ea17fef0dd5d24cb93439ec5af0c7615f58c3f22605"
+checksum = "147100a7bea70fa20ef224a6bad700358305f5dc0f84649c53769761395b355b"
dependencies = [
"base64-simd",
"bytes",
@@ -917,9 +917,9 @@ dependencies = [
[[package]]
name = "cc"
-version = "1.1.21"
+version = "1.1.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "07b1695e2c7e8fc85310cde85aeaab7e3097f593c91d209d3f9df76c928100f0"
+checksum = "2e80e3b6a3ab07840e1cae9b0666a63970dc28e8ed5ffbcdacbfc760c281bfc1"
dependencies = [
"jobserver",
"libc",
@@ -953,9 +953,9 @@ dependencies = [
[[package]]
name = "chrono-tz"
-version = "0.9.0"
+version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "93698b29de5e97ad0ae26447b344c482a7284c737d9ddc5f9e52b74a336671bb"
+checksum = "cd6dd8046d00723a59a2f8c5f295c515b9bb9a331ee4f8f3d4dd49e428acd3b6"
dependencies = [
"chrono",
"chrono-tz-build",
@@ -964,20 +964,19 @@ dependencies = [
[[package]]
name = "chrono-tz-build"
-version = "0.3.0"
+version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0c088aee841df9c3041febbb73934cfc39708749bf96dc827e3359cd39ef11b1"
+checksum = "e94fea34d77a245229e7746bd2beb786cd2a896f306ff491fb8cecb3074b10a7"
dependencies = [
"parse-zoneinfo",
- "phf",
"phf_codegen",
]
[[package]]
name = "clap"
-version = "4.5.17"
+version = "4.5.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3e5a21b8495e732f1b3c364c9949b201ca7bae518c502c80256c96ad79eaf6ac"
+checksum = "7be5744db7978a28d9df86a214130d106a89ce49644cbc4e3f0c22c3fba30615"
dependencies = [
"clap_builder",
"clap_derive",
@@ -985,9 +984,9 @@ dependencies = [
[[package]]
name = "clap_builder"
-version = "4.5.17"
+version = "4.5.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8cf2dd12af7a047ad9d6da2b6b249759a22a7abc0f474c1dae1777afa4b21a73"
+checksum = "a5fbc17d3ef8278f55b282b2a2e75ae6f6c7d4bb70ed3d0382375104bfafdb4b"
dependencies = [
"anstream",
"anstyle",
@@ -997,9 +996,9 @@ dependencies = [
[[package]]
name = "clap_derive"
-version = "4.5.13"
+version = "4.5.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "501d359d5f3dcaf6ecdeee48833ae73ec6e42723a1e52419c79abf9507eec0a0"
+checksum = "4ac6a0c7b1a9e9a5186361f67dfa1b88213572f427fb9ab038efb2bd8c582dab"
dependencies = [
"heck 0.5.0",
"proc-macro2",
@@ -1175,7 +1174,7 @@ checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf"
dependencies = [
"cfg-if",
"crossbeam-utils",
- "hashbrown",
+ "hashbrown 0.14.5",
"lock_api",
"once_cell",
"parking_lot_core",
@@ -1216,7 +1215,7 @@ dependencies = [
"futures",
"glob",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"indexmap",
"itertools",
"log",
@@ -1293,7 +1292,7 @@ dependencies = [
"arrow-schema",
"chrono",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"instant",
"libc",
"num_cpus",
@@ -1322,7 +1321,7 @@ dependencies = [
"datafusion-common",
"datafusion-expr",
"futures",
- "hashbrown",
+ "hashbrown 0.14.5",
"log",
"object_store",
"parking_lot",
@@ -1345,6 +1344,7 @@ dependencies = [
"datafusion-functions-aggregate-common",
"datafusion-functions-window-common",
"datafusion-physical-expr-common",
+ "indexmap",
"paste",
"serde_json",
"sqlparser",
@@ -1374,7 +1374,7 @@ dependencies = [
"datafusion-common",
"datafusion-execution",
"datafusion-expr",
- "hashbrown",
+ "hashbrown 0.14.5",
"hex",
"itertools",
"log",
@@ -1400,9 +1400,9 @@ dependencies = [
"datafusion-physical-expr",
"datafusion-physical-expr-common",
"half",
+ "indexmap",
"log",
"paste",
- "sqlparser",
]
[[package]]
@@ -1447,6 +1447,7 @@ dependencies = [
"datafusion-functions-window-common",
"datafusion-physical-expr-common",
"log",
+ "paste",
]
[[package]]
@@ -1466,7 +1467,7 @@ dependencies = [
"datafusion-common",
"datafusion-expr",
"datafusion-physical-expr",
- "hashbrown",
+ "hashbrown 0.14.5",
"indexmap",
"itertools",
"log",
@@ -1494,7 +1495,7 @@ dependencies = [
"datafusion-functions-aggregate-common",
"datafusion-physical-expr-common",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"hex",
"indexmap",
"itertools",
@@ -1512,7 +1513,7 @@ dependencies = [
"arrow",
"datafusion-common",
"datafusion-expr-common",
- "hashbrown",
+ "hashbrown 0.14.5",
"rand",
]
@@ -1551,7 +1552,7 @@ dependencies = [
"datafusion-physical-expr-common",
"futures",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"indexmap",
"itertools",
"log",
@@ -1722,9 +1723,9 @@ dependencies = [
[[package]]
name = "flate2"
-version = "1.0.33"
+version = "1.0.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "324a1be68054ef05ad64b861cc9eaf1d623d2d8cb25b4bf2cb9cdd902b4bf253"
+checksum = "a1b589b4dc103969ad3cf85c950899926ec64300a1a46d76c03a6072957036f0"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -1756,9 +1757,9 @@ dependencies = [
[[package]]
name = "futures"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "645c6916888f6cb6350d2550b80fb63e734897a8498abe35cfb732b6487804b0"
+checksum = "65bc07b1a8bc7c85c5f2e110c476c7389b4554ba72af57d8445ea63a576b0876"
dependencies = [
"futures-channel",
"futures-core",
@@ -1771,9 +1772,9 @@ dependencies = [
[[package]]
name = "futures-channel"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eac8f7d7865dcb88bd4373ab671c8cf4508703796caa2b1985a9ca867b3fcb78"
+checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10"
dependencies = [
"futures-core",
"futures-sink",
@@ -1781,15 +1782,15 @@ dependencies = [
[[package]]
name = "futures-core"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dfc6580bb841c5a68e9ef15c77ccc837b40a7504914d52e47b8b0e9bbda25a1d"
+checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e"
[[package]]
name = "futures-executor"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a576fc72ae164fca6b9db127eaa9a9dda0d61316034f33a0a0d4eda41f02b01d"
+checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f"
dependencies = [
"futures-core",
"futures-task",
@@ -1798,15 +1799,15 @@ dependencies = [
[[package]]
name = "futures-io"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a44623e20b9681a318efdd71c299b6b222ed6f231972bfe2f224ebad6311f0c1"
+checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6"
[[package]]
name = "futures-macro"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "87750cf4b7a4c0625b1529e4c543c2182106e4dedc60a2a6455e00d212c489ac"
+checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650"
dependencies = [
"proc-macro2",
"quote",
@@ -1815,15 +1816,15 @@ dependencies = [
[[package]]
name = "futures-sink"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9fb8e00e87438d937621c1c6269e53f536c14d3fbd6a042bb24879e57d474fb5"
+checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7"
[[package]]
name = "futures-task"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "38d84fa142264698cdce1a9f9172cf383a0c82de1bddcf3092901442c4097004"
+checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988"
[[package]]
name = "futures-timer"
@@ -1833,9 +1834,9 @@ checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24"
[[package]]
name = "futures-util"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3d6401deb83407ab3da39eba7e33987a73c3df0c82b4bb5813ee871c19c41d48"
+checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81"
dependencies = [
"futures-channel",
"futures-core",
@@ -1872,9 +1873,9 @@ dependencies = [
[[package]]
name = "gimli"
-version = "0.31.0"
+version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "32085ea23f3234fc7846555e85283ba4de91e21016dc0455a16286d87a292d64"
+checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f"
[[package]]
name = "glob"
@@ -1941,6 +1942,12 @@ dependencies = [
"allocator-api2",
]
+[[package]]
+name = "hashbrown"
+version = "0.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e087f84d4f86bf4b218b927129862374b72199ae7d8657835f1e89000eea4fb"
+
[[package]]
name = "heck"
version = "0.4.1"
@@ -2041,9 +2048,9 @@ dependencies = [
[[package]]
name = "httparse"
-version = "1.9.4"
+version = "1.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fcc0b4a115bf80b728eb8ea024ad5bd707b615bfed49e0665b6e0f86fd082d9"
+checksum = "7d71d3574edd2771538b901e6549113b4006ece66150fb69c0fb6d9a2adae946"
[[package]]
name = "httpdate"
@@ -2127,7 +2134,7 @@ dependencies = [
"http 1.1.0",
"hyper 1.4.1",
"hyper-util",
- "rustls 0.23.13",
+ "rustls 0.23.14",
"rustls-native-certs 0.8.0",
"rustls-pki-types",
"tokio",
@@ -2137,9 +2144,9 @@ dependencies = [
[[package]]
name = "hyper-util"
-version = "0.1.8"
+version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "da62f120a8a37763efb0cf8fdf264b884c7b8b9ac8660b900c8661030c00e6ba"
+checksum = "41296eb09f183ac68eec06e03cdbea2e759633d4067b2f6552fc2e009bcad08b"
dependencies = [
"bytes",
"futures-channel",
@@ -2150,7 +2157,6 @@ dependencies = [
"pin-project-lite",
"socket2",
"tokio",
- "tower",
"tower-service",
"tracing",
]
@@ -2190,12 +2196,12 @@ dependencies = [
[[package]]
name = "indexmap"
-version = "2.5.0"
+version = "2.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "68b900aa2f7301e21c36462b170ee99994de34dff39a4a6a528e80e7376d07e5"
+checksum = "707907fe3c25f5424cce2cb7e1cbcafee6bdbe735ca90ef77c29e84591e5b9da"
dependencies = [
"equivalent",
- "hashbrown",
+ "hashbrown 0.15.0",
]
[[package]]
@@ -2218,9 +2224,9 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02"
[[package]]
name = "ipnet"
-version = "2.10.0"
+version = "2.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "187674a687eed5fe42285b40c6291f9a01517d415fad1c3cbc6a9f778af7fcd4"
+checksum = "ddc24109865250148c2e0f3d25d4f0f479571723792d3802153c60922a4fb708"
[[package]]
name = "is_terminal_polyfill"
@@ -2269,9 +2275,9 @@ checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
[[package]]
name = "lexical-core"
-version = "0.8.5"
+version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2cde5de06e8d4c2faabc400238f9ae1c74d5412d03a7bd067645ccbc47070e46"
+checksum = "0431c65b318a590c1de6b8fd6e72798c92291d27762d94c9e6c37ed7a73d8458"
dependencies = [
"lexical-parse-float",
"lexical-parse-integer",
@@ -2282,9 +2288,9 @@ dependencies = [
[[package]]
name = "lexical-parse-float"
-version = "0.8.5"
+version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "683b3a5ebd0130b8fb52ba0bdc718cc56815b6a097e28ae5a6997d0ad17dc05f"
+checksum = "eb17a4bdb9b418051aa59d41d65b1c9be5affab314a872e5ad7f06231fb3b4e0"
dependencies = [
"lexical-parse-integer",
"lexical-util",
@@ -2293,9 +2299,9 @@ dependencies = [
[[package]]
name = "lexical-parse-integer"
-version = "0.8.6"
+version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6d0994485ed0c312f6d965766754ea177d07f9c00c9b82a5ee62ed5b47945ee9"
+checksum = "5df98f4a4ab53bf8b175b363a34c7af608fe31f93cc1fb1bf07130622ca4ef61"
dependencies = [
"lexical-util",
"static_assertions",
@@ -2303,18 +2309,18 @@ dependencies = [
[[package]]
name = "lexical-util"
-version = "0.8.5"
+version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5255b9ff16ff898710eb9eb63cb39248ea8a5bb036bea8085b1a767ff6c4e3fc"
+checksum = "85314db53332e5c192b6bca611fb10c114a80d1b831ddac0af1e9be1b9232ca0"
dependencies = [
"static_assertions",
]
[[package]]
name = "lexical-write-float"
-version = "0.8.5"
+version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "accabaa1c4581f05a3923d1b4cfd124c329352288b7b9da09e766b0668116862"
+checksum = "6e7c3ad4e37db81c1cbe7cf34610340adc09c322871972f74877a712abc6c809"
dependencies = [
"lexical-util",
"lexical-write-integer",
@@ -2323,9 +2329,9 @@ dependencies = [
[[package]]
name = "lexical-write-integer"
-version = "0.8.5"
+version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e1b6f3d1f4422866b68192d62f77bc5c700bee84f3069f2469d7bc8c77852446"
+checksum = "eb89e9f6958b83258afa3deed90b5de9ef68eef090ad5086c791cd2345610162"
dependencies = [
"lexical-util",
"static_assertions",
@@ -2333,9 +2339,9 @@ dependencies = [
[[package]]
name = "libc"
-version = "0.2.158"
+version = "0.2.159"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d8adc4bb1803a324070e64a98ae98f38934d91957a99cfb3a43dcbc01bc56439"
+checksum = "561d97a539a36e26a9a5fad1ea11a3039a67714694aaa379433e580854bc3dc5"
[[package]]
name = "libflate"
@@ -2357,7 +2363,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e0d73b369f386f1c44abd9c570d5318f55ccde816ff4b562fa452e5182863d"
dependencies = [
"core2",
- "hashbrown",
+ "hashbrown 0.14.5",
"rle-decode-fast",
]
@@ -2600,9 +2606,9 @@ dependencies = [
[[package]]
name = "object"
-version = "0.36.4"
+version = "0.36.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "084f1a5821ac4c651660a94a7153d27ac9d8a53736203f58b31945ded098070a"
+checksum = "aedf0a2d09c573ed1d8d85b30c119153926a2b36dce0ab28322c09a117a4683e"
dependencies = [
"memchr",
]
@@ -2628,7 +2634,7 @@ dependencies = [
"rand",
"reqwest",
"ring",
- "rustls-pemfile 2.1.3",
+ "rustls-pemfile 2.2.0",
"serde",
"serde_json",
"snafu",
@@ -2640,9 +2646,9 @@ dependencies = [
[[package]]
name = "once_cell"
-version = "1.19.0"
+version = "1.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
+checksum = "1261fe7e33c73b354eab43b1273a57c8f967d0391e80353e51f764ac02cf6775"
[[package]]
name = "openssl-probe"
@@ -2696,9 +2702,9 @@ dependencies = [
[[package]]
name = "parquet"
-version = "53.0.0"
+version = "53.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f0fbf928021131daaa57d334ca8e3904fe9ae22f73c56244fc7db9b04eedc3d8"
+checksum = "310c46a70a3ba90d98fec39fa2da6d9d731e544191da6fb56c9d199484d0dd3e"
dependencies = [
"ahash",
"arrow-array",
@@ -2715,7 +2721,7 @@ dependencies = [
"flate2",
"futures",
"half",
- "hashbrown",
+ "hashbrown 0.14.5",
"lz4_flex",
"num",
"num-bigint",
@@ -2799,26 +2805,6 @@ dependencies = [
"siphasher",
]
-[[package]]
-name = "pin-project"
-version = "1.1.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b6bf43b791c5b9e34c3d182969b4abb522f9343702850a2e57f460d00d09b4b3"
-dependencies = [
- "pin-project-internal",
-]
-
-[[package]]
-name = "pin-project-internal"
-version = "1.1.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2f38a4412a78282e09a2cf38d195ea5420d15ba0602cb375210efbc877243965"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
-
[[package]]
name = "pin-project-lite"
version = "0.2.14"
@@ -2833,9 +2819,9 @@ checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
[[package]]
name = "pkg-config"
-version = "0.3.30"
+version = "0.3.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d231b230927b5e4ad203db57bbcbee2802f6bce620b1e4a9024a07d94e2907ec"
+checksum = "953ec861398dccce10c670dfeaf3ec4911ca479e9c02154b3a215178c5f566f2"
[[package]]
name = "powerfmt"
@@ -2908,9 +2894,9 @@ checksum = "b76f1009795ca44bb5aaae8fd3f18953e209259c33d9b059b1f53d58ab7511db"
[[package]]
name = "quick-xml"
-version = "0.36.1"
+version = "0.36.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "96a05e2e8efddfa51a84ca47cec303fac86c8541b686d37cac5efc0e094417bc"
+checksum = "f7649a7b4df05aed9ea7ec6f628c67c9953a43869b8bc50929569b2999d443fe"
dependencies = [
"memchr",
"serde",
@@ -2927,7 +2913,7 @@ dependencies = [
"quinn-proto",
"quinn-udp",
"rustc-hash",
- "rustls 0.23.13",
+ "rustls 0.23.14",
"socket2",
"thiserror",
"tokio",
@@ -2944,7 +2930,7 @@ dependencies = [
"rand",
"ring",
"rustc-hash",
- "rustls 0.23.13",
+ "rustls 0.23.14",
"slab",
"thiserror",
"tinyvec",
@@ -3015,9 +3001,9 @@ dependencies = [
[[package]]
name = "redox_syscall"
-version = "0.5.4"
+version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0884ad60e090bf1345b93da0a5de8923c93884cd03f40dfcfddd3b4bee661853"
+checksum = "9b6dfecf2c74bce2466cabf93f6664d6998a69eb21e39f4207930065b27b771f"
dependencies = [
"bitflags 2.6.0",
]
@@ -3035,9 +3021,9 @@ dependencies = [
[[package]]
name = "regex"
-version = "1.10.6"
+version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4219d74c6b67a3654a9fbebc4b419e22126d13d2f3c4a07ee0cb61ff79a79619"
+checksum = "38200e5ee88914975b69f657f0801b6f6dccafd44fd9326302a4aaeecfacb1d8"
dependencies = [
"aho-corasick",
"memchr",
@@ -3047,9 +3033,9 @@ dependencies = [
[[package]]
name = "regex-automata"
-version = "0.4.7"
+version = "0.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "38caf58cc5ef2fed281f89292ef23f6365465ed9a41b7a7754eb4e26496c92df"
+checksum = "368758f23274712b504848e9d5a6f010445cc8b87a7cdb4d7cbee666c1288da3"
dependencies = [
"aho-corasick",
"memchr",
@@ -3064,9 +3050,9 @@ checksum = "53a49587ad06b26609c52e423de037e7f57f20d53535d66e08c695f347df952a"
[[package]]
name = "regex-syntax"
-version = "0.8.4"
+version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7a66a03ae7c801facd77a29370b4faec201768915ac14a721ba36f20bc9c209b"
+checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c"
[[package]]
name = "relative-path"
@@ -3076,9 +3062,9 @@ checksum = "ba39f3699c378cd8970968dcbff9c43159ea4cfbd88d43c00b22f2ef10a435d2"
[[package]]
name = "reqwest"
-version = "0.12.7"
+version = "0.12.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f8f4955649ef5c38cc7f9e8aa41761d48fb9677197daea9984dc54f56aad5e63"
+checksum = "f713147fbe92361e52392c73b8c9e48c04c6625bce969ef54dc901e58e042a7b"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -3099,9 +3085,9 @@ dependencies = [
"percent-encoding",
"pin-project-lite",
"quinn",
- "rustls 0.23.13",
- "rustls-native-certs 0.7.3",
- "rustls-pemfile 2.1.3",
+ "rustls 0.23.14",
+ "rustls-native-certs 0.8.0",
+ "rustls-pemfile 2.2.0",
"rustls-pki-types",
"serde",
"serde_json",
@@ -3218,9 +3204,9 @@ dependencies = [
[[package]]
name = "rustls"
-version = "0.23.13"
+version = "0.23.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f2dabaac7466917e566adb06783a81ca48944c6898a1b08b9374106dd671f4c8"
+checksum = "415d9944693cb90382053259f89fbb077ea730ad7273047ec63b19bc9b160ba8"
dependencies = [
"once_cell",
"ring",
@@ -3242,19 +3228,6 @@ dependencies = [
"security-framework",
]
-[[package]]
-name = "rustls-native-certs"
-version = "0.7.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5bfb394eeed242e909609f56089eecfe5fda225042e8b171791b9c95f5931e5"
-dependencies = [
- "openssl-probe",
- "rustls-pemfile 2.1.3",
- "rustls-pki-types",
- "schannel",
- "security-framework",
-]
-
[[package]]
name = "rustls-native-certs"
version = "0.8.0"
@@ -3262,7 +3235,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fcaf18a4f2be7326cd874a5fa579fae794320a0f388d365dca7e480e55f83f8a"
dependencies = [
"openssl-probe",
- "rustls-pemfile 2.1.3",
+ "rustls-pemfile 2.2.0",
"rustls-pki-types",
"schannel",
"security-framework",
@@ -3279,19 +3252,18 @@ dependencies = [
[[package]]
name = "rustls-pemfile"
-version = "2.1.3"
+version = "2.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "196fe16b00e106300d3e45ecfcb764fa292a535d7326a29a5875c579c7417425"
+checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50"
dependencies = [
- "base64 0.22.1",
"rustls-pki-types",
]
[[package]]
name = "rustls-pki-types"
-version = "1.8.0"
+version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fc0a2ce646f8655401bb81e7927b812614bd5d91dbc968696be50603510fcaf0"
+checksum = "0e696e35370c65c9c541198af4543ccd580cf17fc25d8e05c5a242b202488c55"
[[package]]
name = "rustls-webpki"
@@ -3359,9 +3331,9 @@ dependencies = [
[[package]]
name = "schannel"
-version = "0.1.24"
+version = "0.1.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e9aaafd5a2b6e3d657ff009d82fbd630b6bd54dd4eb06f21693925cdf80f9b8b"
+checksum = "01227be5826fa0690321a2ba6c5cd57a19cf3f6a09e76973b58e61de6ab9d1c1"
dependencies = [
"windows-sys 0.59.0",
]
@@ -3397,9 +3369,9 @@ dependencies = [
[[package]]
name = "security-framework-sys"
-version = "2.11.1"
+version = "2.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "75da29fe9b9b08fe9d6b22b5b4bcbc75d8db3aa31e639aa56bb62e9d46bfceaf"
+checksum = "ea4a292869320c0272d7bc55a5a6aafaff59b4f63404a003887b679a2e05b4b6"
dependencies = [
"core-foundation-sys",
"libc",
@@ -3510,18 +3482,18 @@ checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67"
[[package]]
name = "snafu"
-version = "0.8.4"
+version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2b835cb902660db3415a672d862905e791e54d306c6e8189168c7f3d9ae1c79d"
+checksum = "223891c85e2a29c3fe8fb900c1fae5e69c2e42415e3177752e8718475efa5019"
dependencies = [
"snafu-derive",
]
[[package]]
name = "snafu-derive"
-version = "0.8.4"
+version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "38d1e02fca405f6280643174a50c942219f0bbf4dbf7d480f1dd864d6f211ae5"
+checksum = "03c3c6b7927ffe7ecaa769ee0e3994da3b8cafc8f444578982c83ecb161af917"
dependencies = [
"heck 0.5.0",
"proc-macro2",
@@ -3633,9 +3605,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "syn"
-version = "2.0.77"
+version = "2.0.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9f35bcdf61fd8e7be6caf75f429fdca8beb3ed76584befb503b1569faee373ed"
+checksum = "89132cd0bf050864e1d38dc3bbc07a0eb8e7530af26344d3d2bbbef83499f590"
dependencies = [
"proc-macro2",
"quote",
@@ -3653,9 +3625,9 @@ dependencies = [
[[package]]
name = "tempfile"
-version = "3.12.0"
+version = "3.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04cbcdd0c794ebb0d4cf35e88edd2f7d2c4c3e9a5a6dab322839b321c6a87a64"
+checksum = "f0f2c9fc62d0beef6951ccffd757e241266a2c833136efbe35af6cd2567dca5b"
dependencies = [
"cfg-if",
"fastrand",
@@ -3672,18 +3644,18 @@ checksum = "3369f5ac52d5eb6ab48c6b4ffdc8efbcad6b89c765749064ba298f2c68a16a76"
[[package]]
name = "thiserror"
-version = "1.0.63"
+version = "1.0.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c0342370b38b6a11b6cc11d6a805569958d54cfa061a29969c3b5ce2ea405724"
+checksum = "d50af8abc119fb8bb6dbabcfa89656f46f84aa0ac7688088608076ad2b459a84"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
-version = "1.0.63"
+version = "1.0.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4558b58466b9ad7ca0f102865eccc95938dca1a74a856f2b57b6629050da261"
+checksum = "08904e7672f5eb876eaaf87e0ce17857500934f4981c4a0ab2b4aa98baac7fc3"
dependencies = [
"proc-macro2",
"quote",
@@ -3800,7 +3772,7 @@ version = "0.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c7bc40d0e5a97695bb96e27995cd3a08538541b0a846f65bba7a359f36700d4"
dependencies = [
- "rustls 0.23.13",
+ "rustls 0.23.14",
"rustls-pki-types",
"tokio",
]
@@ -3826,36 +3798,15 @@ checksum = "0dd7358ecb8fc2f8d014bf86f6f638ce72ba252a2c3a2572f2a795f1d23efb41"
[[package]]
name = "toml_edit"
-version = "0.22.21"
+version = "0.22.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b072cee73c449a636ffd6f32bd8de3a9f7119139aff882f44943ce2986dc5cf"
+checksum = "4ae48d6208a266e853d946088ed816055e556cc6028c5e8e2b84d9fa5dd7c7f5"
dependencies = [
"indexmap",
"toml_datetime",
"winnow",
]
-[[package]]
-name = "tower"
-version = "0.4.13"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
-dependencies = [
- "futures-core",
- "futures-util",
- "pin-project",
- "pin-project-lite",
- "tokio",
- "tower-layer",
- "tower-service",
-]
-
-[[package]]
-name = "tower-layer"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
-
[[package]]
name = "tower-service"
version = "0.3.3"
@@ -3937,9 +3888,9 @@ checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825"
[[package]]
name = "unicode-bidi"
-version = "0.3.15"
+version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08f95100a766bf4f8f28f90d77e0a5461bbdb219042e7679bebe79004fed8d75"
+checksum = "5ab17db44d7388991a428b2ee655ce0c212e862eff1768a455c58f9aad6e7893"
[[package]]
name = "unicode-ident"
@@ -3964,9 +3915,9 @@ checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493"
[[package]]
name = "unicode-width"
-version = "0.1.13"
+version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0336d538f7abc86d282a4189614dfaa90810dfc2c6f6427eaf88e16311dd225d"
+checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"
[[package]]
name = "untrusted"
@@ -4122,9 +4073,9 @@ checksum = "c62a0a307cb4a311d3a07867860911ca130c3494e8c2719593806c08bc5d0484"
[[package]]
name = "wasm-streams"
-version = "0.4.0"
+version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b65dc4c90b63b118468cf747d8bf3566c1913ef60be765b5730ead9e0a3ba129"
+checksum = "4e072d4e72f700fb3443d8fe94a39315df013eef1104903cdb0a2abd322bbecd"
dependencies = [
"futures-util",
"js-sys",
@@ -4341,9 +4292,9 @@ checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
[[package]]
name = "winnow"
-version = "0.6.18"
+version = "0.6.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "68a9bda4691f099d435ad181000724da8e5899daa10713c2d432552b9ccd3a6f"
+checksum = "36c1fec1a2bb5866f07c25f68c26e565c4c200aebb96d7e55710c19d3e8ac49b"
dependencies = [
"memchr",
]
diff --git a/datafusion/common/src/dfschema.rs b/datafusion/common/src/dfschema.rs
index 0dec14e917..69cdf866cf 100644
--- a/datafusion/common/src/dfschema.rs
+++ b/datafusion/common/src/dfschema.rs
@@ -226,7 +226,12 @@ impl DFSchema {
for (field, qualifier) in self.inner.fields().iter().zip(&self.field_qualifiers) {
if let Some(qualifier) = qualifier {
- qualified_names.insert((qualifier, field.name()));
+ if !qualified_names.insert((qualifier, field.name())) {
+ return _schema_err!(SchemaError::DuplicateQualifiedField {
+ qualifier: Box::new(qualifier.clone()),
+ name: field.name().to_string(),
+ });
+ }
} else if !unqualified_names.insert(field.name()) {
return _schema_err!(SchemaError::DuplicateUnqualifiedField {
name: field.name().to_string()
@@ -1165,7 +1170,10 @@ mod tests {
let left = DFSchema::try_from_qualified_schema("t1", &test_schema_1())?;
let right = DFSchema::try_from_qualified_schema("t1", &test_schema_1())?;
let join = left.join(&right);
- assert!(join.err().is_none());
+ assert_eq!(
+ join.unwrap_err().strip_backtrace(),
+ "Schema error: Schema contains duplicate qualified field name t1.c0",
+ );
Ok(())
}
diff --git a/datafusion/common/src/tree_node.rs b/datafusion/common/src/tree_node.rs
index 88300e3edd..b4d3251fd2 100644
--- a/datafusion/common/src/tree_node.rs
+++ b/datafusion/common/src/tree_node.rs
@@ -681,7 +681,7 @@ impl<T> Transformed<T> {
}
}
- /// Create a `Transformed` with `transformed and [`TreeNodeRecursion::Continue`].
+ /// Create a `Transformed` with `transformed` and [`TreeNodeRecursion::Continue`].
pub fn new_transformed(data: T, transformed: bool) -> Self {
Self::new(data, transformed, TreeNodeRecursion::Continue)
}
diff --git a/datafusion/core/src/bin/print_functions_docs.rs b/datafusion/core/src/bin/print_functions_docs.rs
new file mode 100644
index 0000000000..53cfe94eca
--- /dev/null
+++ b/datafusion/core/src/bin/print_functions_docs.rs
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion::execution::SessionStateDefaults;
+use datafusion_expr::{
+ aggregate_doc_sections, scalar_doc_sections, window_doc_sections, AggregateUDF,
+ DocSection, Documentation, ScalarUDF, WindowUDF,
+};
+use itertools::Itertools;
+use std::env::args;
+use std::fmt::Write as _;
+
+fn main() {
+ let args: Vec<String> = args().collect();
+
+ if args.len() != 2 {
+ panic!(
+ "Usage: {} type (one of 'aggregate', 'scalar', 'window')",
+ args[0]
+ );
+ }
+
+ let function_type = args[1].trim().to_lowercase();
+ let docs = match function_type.as_str() {
+ "aggregate" => print_aggregate_docs(),
+ "scalar" => print_scalar_docs(),
+ "window" => print_window_docs(),
+ _ => {
+ panic!("Unknown function type: {}", function_type)
+ }
+ };
+
+ println!("{docs}");
+}
+
+fn print_aggregate_docs() -> String {
+ let mut providers: Vec<Box<dyn DocProvider>> = vec![];
+
+ for f in SessionStateDefaults::default_aggregate_functions() {
+ providers.push(Box::new(f.as_ref().clone()));
+ }
+
+ print_docs(providers, aggregate_doc_sections::doc_sections())
+}
+
+fn print_scalar_docs() -> String {
+ let mut providers: Vec<Box<dyn DocProvider>> = vec![];
+
+ for f in SessionStateDefaults::default_scalar_functions() {
+ providers.push(Box::new(f.as_ref().clone()));
+ }
+
+ print_docs(providers, scalar_doc_sections::doc_sections())
+}
+
+fn print_window_docs() -> String {
+ let mut providers: Vec<Box<dyn DocProvider>> = vec![];
+
+ for f in SessionStateDefaults::default_window_functions() {
+ providers.push(Box::new(f.as_ref().clone()));
+ }
+
+ print_docs(providers, window_doc_sections::doc_sections())
+}
+
+fn print_docs(
+ providers: Vec<Box<dyn DocProvider>>,
+ doc_sections: Vec<DocSection>,
+) -> String {
+ let mut docs = "".to_string();
+
+ // doc sections only includes sections that have 'include' == true
+ for doc_section in doc_sections {
+ // make sure there is a function that is in this doc section
+ if !&providers.iter().any(|f| {
+ if let Some(documentation) = f.get_documentation() {
+ documentation.doc_section == doc_section
+ } else {
+ false
+ }
+ }) {
+ continue;
+ }
+
+ let providers: Vec<&Box<dyn DocProvider>> = providers
+ .iter()
+ .filter(|&f| {
+ if let Some(documentation) = f.get_documentation() {
+ documentation.doc_section == doc_section
+ } else {
+ false
+ }
+ })
+ .collect::<Vec<_>>();
+
+ // write out section header
+ let _ = writeln!(docs, "## {} ", doc_section.label);
+
+ if let Some(description) = doc_section.description {
+ let _ = writeln!(docs, "{description}");
+ }
+
+ // names is a sorted list of function names and aliases since we display
+ // both in the documentation
+ let names = get_names_and_aliases(&providers);
+
+ // write out the list of function names and aliases
+ names.iter().for_each(|name| {
+ let _ = writeln!(docs, "- [{name}](#{name})");
+ });
+
+ // write out each function and alias in the order of the sorted name list
+ for name in names {
+ let f = providers
+ .iter()
+ .find(|f| f.get_name() == name || f.get_aliases().contains(&name))
+ .unwrap();
+
+ let aliases = f.get_aliases();
+ let documentation = f.get_documentation();
+
+ // if this name is an alias we need to display what it's an alias of
+ if aliases.contains(&name) {
+ let fname = f.get_name();
+ let _ = writeln!(docs, r#"### `{name}`"#);
+ let _ = writeln!(docs, "_Alias of [{fname}](#{fname})._");
+ continue;
+ }
+
+ // otherwise display the documentation for the function
+ let Some(documentation) = documentation else {
+ unreachable!()
+ };
+
+ // first, the name, description and syntax example
+ let _ = write!(
+ docs,
+ r#"
+### `{}`
+
+{}
+
+```
+{}
+```
+"#,
+ name, documentation.description, documentation.syntax_example
+ );
+
+ // next, arguments
+ if let Some(args) = &documentation.arguments {
+ let _ = writeln!(docs, "#### Arguments\n");
+ for (arg_name, arg_desc) in args {
+ let _ = writeln!(docs, "- **{arg_name}**: {arg_desc}");
+ }
+ }
+
+ // next, sql example if provided
+ if let Some(example) = &documentation.sql_example {
+ let _ = writeln!(
+ docs,
+ r#"
+#### Example
+
+{}
+"#,
+ example
+ );
+ }
+
+ // next, aliases
+ if !f.get_aliases().is_empty() {
+ let _ = writeln!(docs, "#### Aliases");
+
+ for alias in f.get_aliases() {
+ let _ = writeln!(docs, "- {}", alias.replace("_", r#"\_"#));
+ }
+ }
+
+ // finally, any related udfs
+ if let Some(related_udfs) = &documentation.related_udfs {
+ let _ = writeln!(docs, "\n**Related functions**:");
+
+ for related in related_udfs {
+ let _ = writeln!(docs, "- [{related}](#{related})");
+ }
+ }
+ }
+ }
+
+ docs
+}
+
+trait DocProvider {
+ fn get_name(&self) -> String;
+ fn get_aliases(&self) -> Vec<String>;
+ fn get_documentation(&self) -> Option<&Documentation>;
+}
+
+impl DocProvider for AggregateUDF {
+ fn get_name(&self) -> String {
+ self.name().to_string()
+ }
+ fn get_aliases(&self) -> Vec<String> {
+ self.aliases().iter().map(|a| a.to_string()).collect()
+ }
+ fn get_documentation(&self) -> Option<&Documentation> {
+ self.documentation()
+ }
+}
+
+impl DocProvider for ScalarUDF {
+ fn get_name(&self) -> String {
+ self.name().to_string()
+ }
+ fn get_aliases(&self) -> Vec<String> {
+ self.aliases().iter().map(|a| a.to_string()).collect()
+ }
+ fn get_documentation(&self) -> Option<&Documentation> {
+ self.documentation()
+ }
+}
+
+impl DocProvider for WindowUDF {
+ fn get_name(&self) -> String {
+ self.name().to_string()
+ }
+ fn get_aliases(&self) -> Vec<String> {
+ self.aliases().iter().map(|a| a.to_string()).collect()
+ }
+ fn get_documentation(&self) -> Option<&Documentation> {
+ self.documentation()
+ }
+}
+
+#[allow(clippy::borrowed_box)]
+#[allow(clippy::ptr_arg)]
+fn get_names_and_aliases(functions: &Vec<&Box<dyn DocProvider>>) -> Vec<String> {
+ functions
+ .iter()
+ .flat_map(|f| {
+ if f.get_aliases().is_empty() {
+ vec![f.get_name().to_string()]
+ } else {
+ let mut names = vec![f.get_name().to_string()];
+ names.extend(f.get_aliases().iter().cloned());
+ names
+ }
+ })
+ .sorted()
+ .collect_vec()
+}
diff --git a/datafusion/core/src/catalog_common/mod.rs b/datafusion/core/src/catalog_common/mod.rs
index b841437886..85207845a0 100644
--- a/datafusion/core/src/catalog_common/mod.rs
+++ b/datafusion/core/src/catalog_common/mod.rs
@@ -185,9 +185,7 @@ pub fn resolve_table_references(
let _ = s.as_ref().visit(visitor);
}
DFStatement::CreateExternalTable(table) => {
- visitor
- .relations
- .insert(ObjectName(vec![Ident::from(table.name.as_str())]));
+ visitor.relations.insert(table.name.clone());
}
DFStatement::CopyTo(CopyToStatement { source, .. }) => match source {
CopyToSource::Relation(table_name) => {
diff --git a/datafusion/core/src/dataframe/mod.rs b/datafusion/core/src/dataframe/mod.rs
index 70c5075114..67e2a4780d 100644
--- a/datafusion/core/src/dataframe/mod.rs
+++ b/datafusion/core/src/dataframe/mod.rs
@@ -535,9 +535,26 @@ impl DataFrame {
group_expr: Vec<Expr>,
aggr_expr: Vec<Expr>,
) -> Result<DataFrame> {
+ let is_grouping_set = matches!(group_expr.as_slice(), [Expr::GroupingSet(_)]);
+ let aggr_expr_len = aggr_expr.len();
let plan = LogicalPlanBuilder::from(self.plan)
.aggregate(group_expr, aggr_expr)?
.build()?;
+ let plan = if is_grouping_set {
+ let grouping_id_pos = plan.schema().fields().len() - 1 - aggr_expr_len;
+ // For grouping sets we do a project to not expose the internal grouping id
+ let exprs = plan
+ .schema()
+ .columns()
+ .into_iter()
+ .enumerate()
+ .filter(|(idx, _)| *idx != grouping_id_pos)
+ .map(|(_, column)| Expr::Column(column))
+ .collect::<Vec<_>>();
+ LogicalPlanBuilder::from(plan).project(exprs)?.build()?
+ } else {
+ plan
+ };
Ok(DataFrame {
session_state: self.session_state,
plan,
@@ -3380,52 +3397,6 @@ mod tests {
Ok(())
}
- // Table 't1' self join
- // Supplementary test of issue: https://github.com/apache/datafusion/issues/7790
- #[tokio::test]
- async fn with_column_self_join() -> Result<()> {
- let df = test_table().await?.select_columns(&["c1"])?;
- let ctx = SessionContext::new();
-
- ctx.register_table("t1", df.into_view())?;
-
- let df = ctx
- .table("t1")
- .await?
- .join(
- ctx.table("t1").await?,
- JoinType::Inner,
- &["c1"],
- &["c1"],
- None,
- )?
- .sort(vec![
- // make the test deterministic
- col("t1.c1").sort(true, true),
- ])?
- .limit(0, Some(1))?;
-
- let df_results = df.clone().collect().await?;
- assert_batches_sorted_eq!(
- [
- "+----+----+",
- "| c1 | c1 |",
- "+----+----+",
- "| a | a |",
- "+----+----+",
- ],
- &df_results
- );
-
- let actual_err = df.clone().with_column("new_column", lit(true)).unwrap_err();
- let expected_err = "Error during planning: Projections require unique expression names \
- but the expression \"t1.c1\" at position 0 and \"t1.c1\" at position 1 have the same name. \
- Consider aliasing (\"AS\") one of them.";
- assert_eq!(actual_err.strip_backtrace(), expected_err);
-
- Ok(())
- }
-
#[tokio::test]
async fn with_column_renamed() -> Result<()> {
let df = test_table()
diff --git a/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs b/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs
index 3a5d50bba0..98b6702bc3 100644
--- a/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs
+++ b/datafusion/core/src/datasource/avro_to_arrow/arrow_array_reader.rs
@@ -573,7 +573,7 @@ impl<'a, R: Read> AvroArrowArrayReader<'a, R> {
// extract list values, with non-lists converted to Value::Null
let array_item_count = rows
.iter()
- .map(|row| match row {
+ .map(|row| match maybe_resolve_union(row) {
Value::Array(values) => values.len(),
_ => 1,
})
@@ -1643,6 +1643,93 @@ mod test {
assert_batches_eq!(expected, &[batch]);
}
+ #[test]
+ fn test_avro_nullable_struct_array() {
+ let schema = apache_avro::Schema::parse_str(
+ r#"
+ {
+ "type": "record",
+ "name": "r1",
+ "fields": [
+ {
+ "name": "col1",
+ "type": [
+ "null",
+ {
+ "type": "array",
+ "items": {
+ "type": [
+ "null",
+ {
+ "type": "record",
+ "name": "Item",
+ "fields": [
+ {
+ "name": "id",
+ "type": "long"
+ }
+ ]
+ }
+ ]
+ }
+ }
+ ],
+ "default": null
+ }
+ ]
+ }"#,
+ )
+ .unwrap();
+ let jv1 = serde_json::json!({
+ "col1": [
+ {
+ "id": 234
+ },
+ {
+ "id": 345
+ }
+ ]
+ });
+ let r1 = apache_avro::to_value(jv1)
+ .unwrap()
+ .resolve(&schema)
+ .unwrap();
+ let r2 = apache_avro::to_value(serde_json::json!({ "col1": null }))
+ .unwrap()
+ .resolve(&schema)
+ .unwrap();
+
+ let mut w = apache_avro::Writer::new(&schema, vec![]);
+ for _i in 0..5 {
+ w.append(r1.clone()).unwrap();
+ }
+ w.append(r2).unwrap();
+ let bytes = w.into_inner().unwrap();
+
+ let mut reader = ReaderBuilder::new()
+ .read_schema()
+ .with_batch_size(20)
+ .build(std::io::Cursor::new(bytes))
+ .unwrap();
+ let batch = reader.next().unwrap().unwrap();
+ assert_eq!(batch.num_rows(), 6);
+ assert_eq!(batch.num_columns(), 1);
+
+ let expected = [
+ "+------------------------+",
+ "| col1 |",
+ "+------------------------+",
+ "| [{id: 234}, {id: 345}] |",
+ "| [{id: 234}, {id: 345}] |",
+ "| [{id: 234}, {id: 345}] |",
+ "| [{id: 234}, {id: 345}] |",
+ "| [{id: 234}, {id: 345}] |",
+ "| |",
+ "+------------------------+",
+ ];
+ assert_batches_eq!(expected, &[batch]);
+ }
+
#[test]
fn test_avro_iterator() {
let reader = build_reader("alltypes_plain.avro", 5);
diff --git a/datafusion/core/src/datasource/dynamic_file.rs b/datafusion/core/src/datasource/dynamic_file.rs
index 3c409af297..6654d0871c 100644
--- a/datafusion/core/src/datasource/dynamic_file.rs
+++ b/datafusion/core/src/datasource/dynamic_file.rs
@@ -69,11 +69,18 @@ impl UrlTableFactory for DynamicListTableFactory {
.ok_or_else(|| plan_datafusion_err!("get current SessionStore error"))?;
match ListingTableConfig::new(table_url.clone())
- .infer(state)
+ .infer_options(state)
.await
{
- Ok(cfg) => ListingTable::try_new(cfg)
- .map(|table| Some(Arc::new(table) as Arc<dyn TableProvider>)),
+ Ok(cfg) => {
+ let cfg = cfg
+ .infer_partitions_from_path(state)
+ .await?
+ .infer_schema(state)
+ .await?;
+ ListingTable::try_new(cfg)
+ .map(|table| Some(Arc::new(table) as Arc<dyn TableProvider>))
+ }
Err(_) => Ok(None),
}
}
diff --git a/datafusion/core/src/datasource/file_format/csv.rs b/datafusion/core/src/datasource/file_format/csv.rs
index e821fa806f..f235c3b628 100644
--- a/datafusion/core/src/datasource/file_format/csv.rs
+++ b/datafusion/core/src/datasource/file_format/csv.rs
@@ -771,7 +771,7 @@ mod tests {
"c7: Int64",
"c8: Int64",
"c9: Int64",
- "c10: Int64",
+ "c10: Utf8",
"c11: Float64",
"c12: Float64",
"c13: Utf8"
@@ -907,7 +907,7 @@ mod tests {
Field::new("c7", DataType::Int64, true),
Field::new("c8", DataType::Int64, true),
Field::new("c9", DataType::Int64, true),
- Field::new("c10", DataType::Int64, true),
+ Field::new("c10", DataType::Utf8, true),
Field::new("c11", DataType::Float64, true),
Field::new("c12", DataType::Float64, true),
Field::new("c13", DataType::Utf8, true),
diff --git a/datafusion/core/src/datasource/file_format/mod.rs b/datafusion/core/src/datasource/file_format/mod.rs
index 60f2b2dcef..e16986c660 100644
--- a/datafusion/core/src/datasource/file_format/mod.rs
+++ b/datafusion/core/src/datasource/file_format/mod.rs
@@ -241,16 +241,14 @@ pub fn transform_schema_to_view(schema: &Schema) -> Schema {
.fields
.iter()
.map(|field| match field.data_type() {
- DataType::Utf8 | DataType::LargeUtf8 => Arc::new(Field::new(
- field.name(),
- DataType::Utf8View,
- field.is_nullable(),
- )),
- DataType::Binary | DataType::LargeBinary => Arc::new(Field::new(
- field.name(),
- DataType::BinaryView,
- field.is_nullable(),
- )),
+ DataType::Utf8 | DataType::LargeUtf8 => Arc::new(
+ Field::new(field.name(), DataType::Utf8View, field.is_nullable())
+ .with_metadata(field.metadata().to_owned()),
+ ),
+ DataType::Binary | DataType::LargeBinary => Arc::new(
+ Field::new(field.name(), DataType::BinaryView, field.is_nullable())
+ .with_metadata(field.metadata().to_owned()),
+ ),
_ => field.clone(),
})
.collect();
diff --git a/datafusion/core/src/datasource/file_format/parquet.rs b/datafusion/core/src/datasource/file_format/parquet.rs
index 98ae0ce14b..8647b5df90 100644
--- a/datafusion/core/src/datasource/file_format/parquet.rs
+++ b/datafusion/core/src/datasource/file_format/parquet.rs
@@ -20,6 +20,7 @@
use std::any::Any;
use std::fmt;
use std::fmt::Debug;
+use std::ops::Range;
use std::sync::Arc;
use super::write::demux::start_demuxer_task;
@@ -47,7 +48,7 @@ use datafusion_common::file_options::parquet_writer::ParquetWriterOptions;
use datafusion_common::parsers::CompressionTypeVariant;
use datafusion_common::stats::Precision;
use datafusion_common::{
- exec_err, internal_datafusion_err, not_impl_err, DataFusionError, GetExt,
+ internal_datafusion_err, not_impl_err, DataFusionError, GetExt,
DEFAULT_PARQUET_EXTENSION,
};
use datafusion_common_runtime::SpawnedTask;
@@ -60,7 +61,7 @@ use datafusion_physical_expr::PhysicalExpr;
use datafusion_physical_plan::metrics::MetricsSet;
use async_trait::async_trait;
-use bytes::{BufMut, BytesMut};
+use bytes::Bytes;
use hashbrown::HashMap;
use log::debug;
use object_store::buffered::BufWriter;
@@ -71,8 +72,7 @@ use parquet::arrow::arrow_writer::{
use parquet::arrow::{
arrow_to_parquet_schema, parquet_to_arrow_schema, AsyncArrowWriter,
};
-use parquet::file::footer::{decode_footer, decode_metadata};
-use parquet::file::metadata::{ParquetMetaData, RowGroupMetaData};
+use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader, RowGroupMetaData};
use parquet::file::properties::WriterProperties;
use parquet::file::writer::SerializedFileWriter;
use parquet::format::FileMetaData;
@@ -84,10 +84,13 @@ use crate::datasource::physical_plan::parquet::{
can_expr_be_pushed_down_with_schemas, ParquetExecBuilder,
};
use datafusion_physical_expr_common::sort_expr::LexRequirement;
-use futures::{StreamExt, TryStreamExt};
+use futures::future::BoxFuture;
+use futures::{FutureExt, StreamExt, TryStreamExt};
use object_store::path::Path;
use object_store::{ObjectMeta, ObjectStore};
use parquet::arrow::arrow_reader::statistics::StatisticsConverter;
+use parquet::arrow::async_reader::MetadataFetch;
+use parquet::errors::ParquetError;
/// Initial writing buffer size. Note this is just a size hint for efficiency. It
/// will grow beyond the set value if needed.
@@ -441,6 +444,33 @@ impl FileFormat for ParquetFormat {
}
}
+/// [`MetadataFetch`] adapter for reading bytes from an [`ObjectStore`]
+struct ObjectStoreFetch<'a> {
+ store: &'a dyn ObjectStore,
+ meta: &'a ObjectMeta,
+}
+
+impl<'a> ObjectStoreFetch<'a> {
+ fn new(store: &'a dyn ObjectStore, meta: &'a ObjectMeta) -> Self {
+ Self { store, meta }
+ }
+}
+
+impl<'a> MetadataFetch for ObjectStoreFetch<'a> {
+ fn fetch(
+ &mut self,
+ range: Range<usize>,
+ ) -> BoxFuture<'_, Result<Bytes, ParquetError>> {
+ async {
+ self.store
+ .get_range(&self.meta.location, range)
+ .await
+ .map_err(ParquetError::from)
+ }
+ .boxed()
+ }
+}
+
/// Fetches parquet metadata from ObjectStore for given object
///
/// This component is a subject to **change** in near future and is exposed for low level integrations
@@ -452,57 +482,14 @@ pub async fn fetch_parquet_metadata(
meta: &ObjectMeta,
size_hint: Option<usize>,
) -> Result<ParquetMetaData> {
- if meta.size < 8 {
- return exec_err!("file size of {} is less than footer", meta.size);
- }
-
- // If a size hint is provided, read more than the minimum size
- // to try and avoid a second fetch.
- let footer_start = if let Some(size_hint) = size_hint {
- meta.size.saturating_sub(size_hint)
- } else {
- meta.size - 8
- };
-
- let suffix = store
- .get_range(&meta.location, footer_start..meta.size)
- .await?;
-
- let suffix_len = suffix.len();
-
- let mut footer = [0; 8];
- footer.copy_from_slice(&suffix[suffix_len - 8..suffix_len]);
-
- let length = decode_footer(&footer)?;
+ let file_size = meta.size;
+ let fetch = ObjectStoreFetch::new(store, meta);
- if meta.size < length + 8 {
- return exec_err!(
- "file size of {} is less than footer + metadata {}",
- meta.size,
- length + 8
- );
- }
-
- // Did not fetch the entire file metadata in the initial read, need to make a second request
- if length > suffix_len - 8 {
- let metadata_start = meta.size - length - 8;
- let remaining_metadata = store
- .get_range(&meta.location, metadata_start..footer_start)
- .await?;
-
- let mut metadata = BytesMut::with_capacity(length);
-
- metadata.put(remaining_metadata.as_ref());
- metadata.put(&suffix[..suffix_len - 8]);
-
- Ok(decode_metadata(metadata.as_ref())?)
- } else {
- let metadata_start = meta.size - length - 8;
-
- Ok(decode_metadata(
- &suffix[metadata_start - footer_start..suffix_len - 8],
- )?)
- }
+ ParquetMetaDataReader::new()
+ .with_prefetch_hint(size_hint)
+ .load_and_finish(fetch, file_size)
+ .await
+ .map_err(DataFusionError::from)
}
/// Read and parse the schema of the Parquet file at location `path`
diff --git a/datafusion/core/src/datasource/listing/table.rs b/datafusion/core/src/datasource/listing/table.rs
index 3eb8eed9de..a9c6aec175 100644
--- a/datafusion/core/src/datasource/listing/table.rs
+++ b/datafusion/core/src/datasource/listing/table.rs
@@ -33,7 +33,7 @@ use crate::datasource::{
};
use crate::execution::context::SessionState;
use datafusion_catalog::TableProvider;
-use datafusion_common::{DataFusionError, Result};
+use datafusion_common::{config_err, DataFusionError, Result};
use datafusion_expr::dml::InsertOp;
use datafusion_expr::{utils::conjunction, Expr, TableProviderFilterPushDown};
use datafusion_expr::{SortExpr, TableType};
@@ -192,6 +192,38 @@ impl ListingTableConfig {
pub async fn infer(self, state: &SessionState) -> Result<Self> {
self.infer_options(state).await?.infer_schema(state).await
}
+
+ /// Infer the partition columns from the path. Requires `self.options` to be set prior to using.
+ pub async fn infer_partitions_from_path(self, state: &SessionState) -> Result<Self> {
+ match self.options {
+ Some(options) => {
+ let Some(url) = self.table_paths.first() else {
+ return config_err!("No table path found");
+ };
+ let partitions = options
+ .infer_partitions(state, url)
+ .await?
+ .into_iter()
+ .map(|col_name| {
+ (
+ col_name,
+ DataType::Dictionary(
+ Box::new(DataType::UInt16),
+ Box::new(DataType::Utf8),
+ ),
+ )
+ })
+ .collect::<Vec<_>>();
+ let options = options.with_table_partition_cols(partitions);
+ Ok(Self {
+ table_paths: self.table_paths,
+ file_schema: self.file_schema,
+ options: Some(options),
+ })
+ }
+ None => config_err!("No `ListingOptions` set for inferring schema"),
+ }
+ }
}
/// Options for creating a [`ListingTable`]
@@ -505,7 +537,7 @@ impl ListingOptions {
/// Infer the partitioning at the given path on the provided object store.
/// For performance reasons, it doesn't read all the files on disk
/// and therefore may fail to detect invalid partitioning.
- async fn infer_partitions(
+ pub(crate) async fn infer_partitions(
&self,
state: &SessionState,
table_path: &ListingTableUrl,
diff --git a/datafusion/core/src/physical_planner.rs b/datafusion/core/src/physical_planner.rs
index 520392c9f0..cf2a157b04 100644
--- a/datafusion/core/src/physical_planner.rs
+++ b/datafusion/core/src/physical_planner.rs
@@ -692,10 +692,6 @@ impl DefaultPhysicalPlanner {
physical_input_schema.clone(),
)?);
- // update group column indices based on partial aggregate plan evaluation
- let final_group: Vec<Arc<dyn PhysicalExpr>> =
- initial_aggr.output_group_expr();
-
let can_repartition = !groups.is_empty()
&& session_state.config().target_partitions() > 1
&& session_state.config().repartition_aggregations();
@@ -716,13 +712,7 @@ impl DefaultPhysicalPlanner {
AggregateMode::Final
};
- let final_grouping_set = PhysicalGroupBy::new_single(
- final_group
- .iter()
- .enumerate()
- .map(|(i, expr)| (expr.clone(), groups.expr()[i].1.clone()))
- .collect(),
- );
+ let final_grouping_set = initial_aggr.group_expr().as_final();
Arc::new(AggregateExec::try_new(
next_partition_mode,
@@ -2345,7 +2335,7 @@ mod tests {
.expect("hash aggregate");
assert_eq!(
"sum(aggregate_test_100.c3)",
- final_hash_agg.schema().field(2).name()
+ final_hash_agg.schema().field(3).name()
);
// we need access to the input to the partial aggregate so that other projects can
// implement serde
@@ -2557,6 +2547,10 @@ mod tests {
) -> Result<Self> {
unimplemented!("NoOp");
}
+
+ fn supports_limit_pushdown(&self) -> bool {
+ false // Disallow limit push-down by default
+ }
}
#[derive(Debug)]
diff --git a/datafusion/core/tests/expr_api/mod.rs b/datafusion/core/tests/expr_api/mod.rs
index cbd8926721..81a3336100 100644
--- a/datafusion/core/tests/expr_api/mod.rs
+++ b/datafusion/core/tests/expr_api/mod.rs
@@ -37,14 +37,14 @@ mod simplification;
fn test_octet_length() {
#[rustfmt::skip]
evaluate_expr_test(
- octet_length(col("list")),
+ octet_length(col("id")),
vec![
"+------+",
"| expr |",
"+------+",
- "| 5 |",
- "| 18 |",
- "| 6 |",
+ "| 1 |",
+ "| 1 |",
+ "| 1 |",
"+------+",
],
);
diff --git a/datafusion/core/tests/fuzz_cases/sort_preserving_repartition_fuzz.rs b/datafusion/core/tests/fuzz_cases/sort_preserving_repartition_fuzz.rs
index 408cadc35f..0cd702372f 100644
--- a/datafusion/core/tests/fuzz_cases/sort_preserving_repartition_fuzz.rs
+++ b/datafusion/core/tests/fuzz_cases/sort_preserving_repartition_fuzz.rs
@@ -29,7 +29,7 @@ mod sp_repartition_fuzz_tests {
metrics::{BaselineMetrics, ExecutionPlanMetricsSet},
repartition::RepartitionExec,
sorts::sort_preserving_merge::SortPreservingMergeExec,
- sorts::streaming_merge::streaming_merge,
+ sorts::streaming_merge::StreamingMergeBuilder,
stream::RecordBatchStreamAdapter,
ExecutionPlan, Partitioning,
};
@@ -246,15 +246,14 @@ mod sp_repartition_fuzz_tests {
MemoryConsumer::new("test".to_string()).register(context.memory_pool());
// Internally SortPreservingMergeExec uses this function for merging.
- let res = streaming_merge(
- streams,
- schema,
- &exprs,
- BaselineMetrics::new(&ExecutionPlanMetricsSet::new(), 0),
- 1,
- None,
- mem_reservation,
- )?;
+ let res = StreamingMergeBuilder::new()
+ .with_streams(streams)
+ .with_schema(schema)
+ .with_expressions(&exprs)
+ .with_metrics(BaselineMetrics::new(&ExecutionPlanMetricsSet::new(), 0))
+ .with_batch_size(1)
+ .with_reservation(mem_reservation)
+ .build()?;
let res = collect(res).await?;
// Contains the merged result.
let res = concat_batches(&res[0].schema(), &res)?;
diff --git a/datafusion/core/tests/user_defined/user_defined_plan.rs b/datafusion/core/tests/user_defined/user_defined_plan.rs
index e51adbc4dd..2b45d0ed60 100644
--- a/datafusion/core/tests/user_defined/user_defined_plan.rs
+++ b/datafusion/core/tests/user_defined/user_defined_plan.rs
@@ -443,6 +443,10 @@ impl UserDefinedLogicalNodeCore for TopKPlanNode {
expr: replace_sort_expression(self.expr.clone(), exprs.swap_remove(0)),
})
}
+
+ fn supports_limit_pushdown(&self) -> bool {
+ false // Disallow limit push-down by default
+ }
}
/// Physical planner for TopK nodes
diff --git a/datafusion/expr-common/src/signature.rs b/datafusion/expr-common/src/signature.rs
index d1553b3315..320e1303a2 100644
--- a/datafusion/expr-common/src/signature.rs
+++ b/datafusion/expr-common/src/signature.rs
@@ -125,6 +125,11 @@ pub enum TypeSignature {
/// Fixed number of arguments of numeric types.
/// See <https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#method.is_numeric> to know which type is considered numeric
Numeric(usize),
+ /// Fixed number of arguments of all the same string types.
+ /// The precedence of type from high to low is Utf8View, LargeUtf8 and Utf8.
+ /// Null is considerd as Utf8 by default
+ /// Dictionary with string value type is also handled.
+ String(usize),
}
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Hash)]
@@ -190,8 +195,11 @@ impl TypeSignature {
.collect::<Vec<String>>()
.join(", ")]
}
+ TypeSignature::String(num) => {
+ vec![format!("String({num})")]
+ }
TypeSignature::Numeric(num) => {
- vec![format!("Numeric({})", num)]
+ vec![format!("Numeric({num})")]
}
TypeSignature::Exact(types) | TypeSignature::Coercible(types) => {
vec![Self::join_types(types, ", ")]
@@ -280,6 +288,14 @@ impl Signature {
}
}
+ /// A specified number of numeric arguments
+ pub fn string(arg_count: usize, volatility: Volatility) -> Self {
+ Self {
+ type_signature: TypeSignature::String(arg_count),
+ volatility,
+ }
+ }
+
/// An arbitrary number of arguments of any type.
pub fn variadic_any(volatility: Volatility) -> Self {
Self {
diff --git a/datafusion/expr-common/src/type_coercion/binary.rs b/datafusion/expr-common/src/type_coercion/binary.rs
index e66a9ae1ea..6d66b8b4df 100644
--- a/datafusion/expr-common/src/type_coercion/binary.rs
+++ b/datafusion/expr-common/src/type_coercion/binary.rs
@@ -471,10 +471,16 @@ fn type_union_resolution_coercion(
let new_value_type = type_union_resolution_coercion(value_type, other_type);
new_value_type.map(|t| DataType::Dictionary(index_type.clone(), Box::new(t)))
}
+ (DataType::List(lhs), DataType::List(rhs)) => {
+ let new_item_type =
+ type_union_resolution_coercion(lhs.data_type(), rhs.data_type());
+ new_item_type.map(|t| DataType::List(Arc::new(Field::new("item", t, true))))
+ }
_ => {
// numeric coercion is the same as comparison coercion, both find the narrowest type
// that can accommodate both types
binary_numeric_coercion(lhs_type, rhs_type)
+ .or_else(|| temporal_coercion_nonstrict_timezone(lhs_type, rhs_type))
.or_else(|| string_coercion(lhs_type, rhs_type))
.or_else(|| numeric_string_coercion(lhs_type, rhs_type))
}
@@ -507,22 +513,6 @@ pub fn comparison_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<D
.or_else(|| struct_coercion(lhs_type, rhs_type))
}
-/// Coerce `lhs_type` and `rhs_type` to a common type for `VALUES` expression
-///
-/// For example `VALUES (1, 2), (3.0, 4.0)` where the first row is `Int32` and
-/// the second row is `Float64` will coerce to `Float64`
-///
-pub fn values_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
- if lhs_type == rhs_type {
- // same type => equality is possible
- return Some(lhs_type.clone());
- }
- binary_numeric_coercion(lhs_type, rhs_type)
- .or_else(|| temporal_coercion_nonstrict_timezone(lhs_type, rhs_type))
- .or_else(|| string_coercion(lhs_type, rhs_type))
- .or_else(|| binary_coercion(lhs_type, rhs_type))
-}
-
/// Coerce `lhs_type` and `rhs_type` to a common type for the purposes of a comparison operation
/// where one is numeric and one is `Utf8`/`LargeUtf8`.
fn string_numeric_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
@@ -969,7 +959,7 @@ fn string_concat_internal_coercion(
/// based on the observation that StringArray to StringViewArray is cheap but not vice versa.
///
/// Between Utf8 and LargeUtf8, we coerce to LargeUtf8.
-fn string_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
+pub fn string_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
use arrow::datatypes::DataType::*;
match (lhs_type, rhs_type) {
// If Utf8View is in any side, we coerce to Utf8View.
diff --git a/datafusion/expr/Cargo.toml b/datafusion/expr/Cargo.toml
index 55387fea22..d7dc1afe4d 100644
--- a/datafusion/expr/Cargo.toml
+++ b/datafusion/expr/Cargo.toml
@@ -48,6 +48,7 @@ datafusion-expr-common = { workspace = true }
datafusion-functions-aggregate-common = { workspace = true }
datafusion-functions-window-common = { workspace = true }
datafusion-physical-expr-common = { workspace = true }
+indexmap = { workspace = true }
paste = "^1.0"
serde_json = { workspace = true }
sqlparser = { workspace = true }
diff --git a/datafusion/expr/src/lib.rs b/datafusion/expr/src/lib.rs
index ff9c37559d..014b271453 100644
--- a/datafusion/expr/src/lib.rs
+++ b/datafusion/expr/src/lib.rs
@@ -34,6 +34,7 @@ mod partition_evaluator;
mod table_source;
mod udaf;
mod udf;
+mod udf_docs;
mod udwf;
pub mod conditional_expressions;
@@ -91,9 +92,12 @@ pub use logical_plan::*;
pub use partition_evaluator::PartitionEvaluator;
pub use sqlparser;
pub use table_source::{TableProviderFilterPushDown, TableSource, TableType};
-pub use udaf::{AggregateUDF, AggregateUDFImpl, ReversedUDAF, StatisticsArgs};
-pub use udf::{ScalarUDF, ScalarUDFImpl};
-pub use udwf::{ReversedUDWF, WindowUDF, WindowUDFImpl};
+pub use udaf::{
+ aggregate_doc_sections, AggregateUDF, AggregateUDFImpl, ReversedUDAF, StatisticsArgs,
+};
+pub use udf::{scalar_doc_sections, ScalarUDF, ScalarUDFImpl};
+pub use udf_docs::{DocSection, Documentation, DocumentationBuilder};
+pub use udwf::{window_doc_sections, ReversedUDWF, WindowUDF, WindowUDFImpl};
pub use window_frame::{WindowFrame, WindowFrameBound, WindowFrameUnits};
#[cfg(test)]
diff --git a/datafusion/expr/src/logical_plan/builder.rs b/datafusion/expr/src/logical_plan/builder.rs
index cc8ddf8ec8..da2a96327c 100644
--- a/datafusion/expr/src/logical_plan/builder.rs
+++ b/datafusion/expr/src/logical_plan/builder.rs
@@ -35,7 +35,6 @@ use crate::logical_plan::{
Projection, Repartition, Sort, SubqueryAlias, TableScan, Union, Unnest, Values,
Window,
};
-use crate::type_coercion::binary::values_coercion;
use crate::utils::{
can_hash, columnize_expr, compare_sort_expr, expr_to_columns,
find_valid_equijoin_key_pair, group_window_expr_by_sort_keys,
@@ -53,6 +52,7 @@ use datafusion_common::{
plan_err, Column, DFSchema, DFSchemaRef, DataFusionError, Result, ScalarValue,
TableReference, ToDFSchema, UnnestOptions,
};
+use datafusion_expr_common::type_coercion::binary::type_union_resolution;
use super::dml::InsertOp;
use super::plan::{ColumnUnnestList, ColumnUnnestType};
@@ -209,7 +209,8 @@ impl LogicalPlanBuilder {
}
if let Some(prev_type) = common_type {
// get common type of each column values.
- let Some(new_type) = values_coercion(&data_type, &prev_type) else {
+ let data_types = vec![prev_type.clone(), data_type.clone()];
+ let Some(new_type) = type_union_resolution(&data_types) else {
return plan_err!("Inconsistent data type across values list at row {i} column {j}. Was {prev_type} but found {data_type}");
};
common_type = Some(new_type);
diff --git a/datafusion/expr/src/logical_plan/extension.rs b/datafusion/expr/src/logical_plan/extension.rs
index d49c85fb6f..19d4cb3db9 100644
--- a/datafusion/expr/src/logical_plan/extension.rs
+++ b/datafusion/expr/src/logical_plan/extension.rs
@@ -195,6 +195,16 @@ pub trait UserDefinedLogicalNode: fmt::Debug + Send + Sync {
/// directly because it must remain object safe.
fn dyn_eq(&self, other: &dyn UserDefinedLogicalNode) -> bool;
fn dyn_ord(&self, other: &dyn UserDefinedLogicalNode) -> Option<Ordering>;
+
+ /// Returns `true` if a limit can be safely pushed down through this
+ /// `UserDefinedLogicalNode` node.
+ ///
+ /// If this method returns `true`, and the query plan contains a limit at
+ /// the output of this node, DataFusion will push the limit to the input
+ /// of this node.
+ fn supports_limit_pushdown(&self) -> bool {
+ false
+ }
}
impl Hash for dyn UserDefinedLogicalNode {
@@ -295,6 +305,16 @@ pub trait UserDefinedLogicalNodeCore:
) -> Option<Vec<Vec<usize>>> {
None
}
+
+ /// Returns `true` if a limit can be safely pushed down through this
+ /// `UserDefinedLogicalNode` node.
+ ///
+ /// If this method returns `true`, and the query plan contains a limit at
+ /// the output of this node, DataFusion will push the limit to the input
+ /// of this node.
+ fn supports_limit_pushdown(&self) -> bool {
+ false // Disallow limit push-down by default
+ }
}
/// Automatically derive UserDefinedLogicalNode to `UserDefinedLogicalNode`
@@ -361,6 +381,10 @@ impl<T: UserDefinedLogicalNodeCore> UserDefinedLogicalNode for T {
.downcast_ref::<Self>()
.and_then(|other| self.partial_cmp(other))
}
+
+ fn supports_limit_pushdown(&self) -> bool {
+ self.supports_limit_pushdown()
+ }
}
fn get_all_columns_from_schema(schema: &DFSchema) -> HashSet<String> {
diff --git a/datafusion/expr/src/logical_plan/plan.rs b/datafusion/expr/src/logical_plan/plan.rs
index 443d23804a..0292274e57 100644
--- a/datafusion/expr/src/logical_plan/plan.rs
+++ b/datafusion/expr/src/logical_plan/plan.rs
@@ -21,7 +21,7 @@ use std::cmp::Ordering;
use std::collections::{HashMap, HashSet};
use std::fmt::{self, Debug, Display, Formatter};
use std::hash::{Hash, Hasher};
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use super::dml::CopyTo;
use super::DdlStatement;
@@ -51,6 +51,7 @@ use datafusion_common::{
DFSchema, DFSchemaRef, DataFusionError, Dependency, FunctionalDependence,
FunctionalDependencies, ParamValues, Result, TableReference, UnnestOptions,
};
+use indexmap::IndexSet;
// backwards compatibility
use crate::display::PgJsonVisitor;
@@ -2964,6 +2965,15 @@ impl Aggregate {
.into_iter()
.map(|(q, f)| (q, f.as_ref().clone().with_nullable(true).into()))
.collect::<Vec<_>>();
+ qualified_fields.push((
+ None,
+ Field::new(
+ Self::INTERNAL_GROUPING_ID,
+ Self::grouping_id_type(qualified_fields.len()),
+ false,
+ )
+ .into(),
+ ));
}
qualified_fields.extend(exprlist_to_fields(aggr_expr.as_slice(), &input)?);
@@ -3015,9 +3025,19 @@ impl Aggregate {
})
}
+ fn is_grouping_set(&self) -> bool {
+ matches!(self.group_expr.as_slice(), [Expr::GroupingSet(_)])
+ }
+
/// Get the output expressions.
fn output_expressions(&self) -> Result<Vec<&Expr>> {
+ static INTERNAL_ID_EXPR: OnceLock<Expr> = OnceLock::new();
let mut exprs = grouping_set_to_exprlist(self.group_expr.as_slice())?;
+ if self.is_grouping_set() {
+ exprs.push(INTERNAL_ID_EXPR.get_or_init(|| {
+ Expr::Column(Column::from_name(Self::INTERNAL_GROUPING_ID))
+ }));
+ }
exprs.extend(self.aggr_expr.iter());
debug_assert!(exprs.len() == self.schema.fields().len());
Ok(exprs)
@@ -3029,6 +3049,41 @@ impl Aggregate {
pub fn group_expr_len(&self) -> Result<usize> {
grouping_set_expr_count(&self.group_expr)
}
+
+ /// Returns the data type of the grouping id.
+ /// The grouping ID value is a bitmask where each set bit
+ /// indicates that the corresponding grouping expression is
+ /// null
+ pub fn grouping_id_type(group_exprs: usize) -> DataType {
+ if group_exprs <= 8 {
+ DataType::UInt8
+ } else if group_exprs <= 16 {
+ DataType::UInt16
+ } else if group_exprs <= 32 {
+ DataType::UInt32
+ } else {
+ DataType::UInt64
+ }
+ }
+
+ /// Internal column used when the aggregation is a grouping set.
+ ///
+ /// This column contains a bitmask where each bit represents a grouping
+ /// expression. The least significant bit corresponds to the rightmost
+ /// grouping expression. A bit value of 0 indicates that the corresponding
+ /// column is included in the grouping set, while a value of 1 means it is excluded.
+ ///
+ /// For example, for the grouping expressions CUBE(a, b), the grouping ID
+ /// column will have the following values:
+ /// 0b00: Both `a` and `b` are included
+ /// 0b01: `b` is excluded
+ /// 0b10: `a` is excluded
+ /// 0b11: Both `a` and `b` are excluded
+ ///
+ /// This internal column is necessary because excluded columns are replaced
+ /// with `NULL` values. To handle these cases correctly, we must distinguish
+ /// between an actual `NULL` value in a column and a column being excluded from the set.
+ pub const INTERNAL_GROUPING_ID: &'static str = "__grouping_id";
}
// Manual implementation needed because of `schema` field. Comparison excludes this field.
@@ -3071,6 +3126,8 @@ fn calc_func_dependencies_for_aggregate(
let group_by_expr_names = group_expr
.iter()
.map(|item| item.schema_name().to_string())
+ .collect::<IndexSet<_>>()
+ .into_iter()
.collect::<Vec<_>>();
let aggregate_func_dependencies = aggregate_functional_dependencies(
input.schema(),
diff --git a/datafusion/expr/src/type_coercion/functions.rs b/datafusion/expr/src/type_coercion/functions.rs
index d30d202df0..143e00fa40 100644
--- a/datafusion/expr/src/type_coercion/functions.rs
+++ b/datafusion/expr/src/type_coercion/functions.rs
@@ -26,8 +26,9 @@ use datafusion_common::{
utils::{coerced_fixed_size_list_to_list, list_ndims},
Result,
};
-use datafusion_expr_common::signature::{
- ArrayFunctionSignature, FIXED_SIZE_LIST_WILDCARD, TIMEZONE_WILDCARD,
+use datafusion_expr_common::{
+ signature::{ArrayFunctionSignature, FIXED_SIZE_LIST_WILDCARD, TIMEZONE_WILDCARD},
+ type_coercion::binary::string_coercion,
};
use std::sync::Arc;
@@ -167,6 +168,21 @@ pub fn data_types(
try_coerce_types(valid_types, current_types, &signature.type_signature)
}
+fn is_well_supported_signature(type_signature: &TypeSignature) -> bool {
+ if let TypeSignature::OneOf(signatures) = type_signature {
+ return signatures.iter().all(is_well_supported_signature);
+ }
+
+ matches!(
+ type_signature,
+ TypeSignature::UserDefined
+ | TypeSignature::Numeric(_)
+ | TypeSignature::String(_)
+ | TypeSignature::Coercible(_)
+ | TypeSignature::Any(_)
+ )
+}
+
fn try_coerce_types(
valid_types: Vec<Vec<DataType>>,
current_types: &[DataType],
@@ -175,14 +191,7 @@ fn try_coerce_types(
let mut valid_types = valid_types;
// Well-supported signature that returns exact valid types.
- if !valid_types.is_empty()
- && matches!(
- type_signature,
- TypeSignature::UserDefined
- | TypeSignature::Numeric(_)
- | TypeSignature::Coercible(_)
- )
- {
+ if !valid_types.is_empty() && is_well_supported_signature(type_signature) {
// exact valid types
assert_eq!(valid_types.len(), 1);
let valid_types = valid_types.swap_remove(0);
@@ -374,6 +383,67 @@ fn get_valid_types(
.iter()
.map(|valid_type| current_types.iter().map(|_| valid_type.clone()).collect())
.collect(),
+ TypeSignature::String(number) => {
+ if *number < 1 {
+ return plan_err!(
+ "The signature expected at least one argument but received {}",
+ current_types.len()
+ );
+ }
+ if *number != current_types.len() {
+ return plan_err!(
+ "The signature expected {} arguments but received {}",
+ number,
+ current_types.len()
+ );
+ }
+
+ fn coercion_rule(
+ lhs_type: &DataType,
+ rhs_type: &DataType,
+ ) -> Result<DataType> {
+ match (lhs_type, rhs_type) {
+ (DataType::Null, DataType::Null) => Ok(DataType::Utf8),
+ (DataType::Null, data_type) | (data_type, DataType::Null) => {
+ coercion_rule(data_type, &DataType::Utf8)
+ }
+ (DataType::Dictionary(_, lhs), DataType::Dictionary(_, rhs)) => {
+ coercion_rule(lhs, rhs)
+ }
+ (DataType::Dictionary(_, v), other)
+ | (other, DataType::Dictionary(_, v)) => coercion_rule(v, other),
+ _ => {
+ if let Some(coerced_type) = string_coercion(lhs_type, rhs_type) {
+ Ok(coerced_type)
+ } else {
+ plan_err!(
+ "{} and {} are not coercible to a common string type",
+ lhs_type,
+ rhs_type
+ )
+ }
+ }
+ }
+ }
+
+ // Length checked above, safe to unwrap
+ let mut coerced_type = current_types.first().unwrap().to_owned();
+ for t in current_types.iter().skip(1) {
+ coerced_type = coercion_rule(&coerced_type, t)?;
+ }
+
+ fn base_type_or_default_type(data_type: &DataType) -> DataType {
+ if data_type.is_null() {
+ DataType::Utf8
+ } else if let DataType::Dictionary(_, v) = data_type {
+ base_type_or_default_type(v)
+ } else {
+ data_type.to_owned()
+ }
+ }
+
+ vec![vec![base_type_or_default_type(&coerced_type); *number]]
+ }
TypeSignature::Numeric(number) => {
if *number < 1 {
return plan_err!(
@@ -602,89 +672,48 @@ fn coerced_from<'a>(
Some(type_into.clone())
}
// coerced into type_into
- (Int8, _) if matches!(type_from, Null | Int8) => Some(type_into.clone()),
- (Int16, _) if matches!(type_from, Null | Int8 | Int16 | UInt8) => {
- Some(type_into.clone())
- }
- (Int32, _)
- if matches!(type_from, Null | Int8 | Int16 | Int32 | UInt8 | UInt16) =>
- {
- Some(type_into.clone())
- }
- (Int64, _)
- if matches!(
- type_from,
- Null | Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32
- ) =>
- {
- Some(type_into.clone())
- }
- (UInt8, _) if matches!(type_from, Null | UInt8) => Some(type_into.clone()),
- (UInt16, _) if matches!(type_from, Null | UInt8 | UInt16) => {
- Some(type_into.clone())
- }
- (UInt32, _) if matches!(type_from, Null | UInt8 | UInt16 | UInt32) => {
- Some(type_into.clone())
- }
- (UInt64, _) if matches!(type_from, Null | UInt8 | UInt16 | UInt32 | UInt64) => {
- Some(type_into.clone())
- }
- (Float32, _)
- if matches!(
- type_from,
- Null | Int8
- | Int16
- | Int32
- | Int64
- | UInt8
- | UInt16
- | UInt32
- | UInt64
- | Float32
- ) =>
- {
- Some(type_into.clone())
- }
- (Float64, _)
- if matches!(
- type_from,
- Null | Int8
- | Int16
- | Int32
- | Int64
- | UInt8
- | UInt16
- | UInt32
- | UInt64
- | Float32
- | Float64
- | Decimal128(_, _)
- ) =>
- {
- Some(type_into.clone())
- }
- (Timestamp(TimeUnit::Nanosecond, None), _)
- if matches!(
- type_from,
- Null | Timestamp(_, None) | Date32 | Utf8 | LargeUtf8
- ) =>
- {
- Some(type_into.clone())
- }
- (Interval(_), _) if matches!(type_from, Utf8 | LargeUtf8) => {
+ (Int8, Null | Int8) => Some(type_into.clone()),
+ (Int16, Null | Int8 | Int16 | UInt8) => Some(type_into.clone()),
+ (Int32, Null | Int8 | Int16 | Int32 | UInt8 | UInt16) => Some(type_into.clone()),
+ (Int64, Null | Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32) => {
Some(type_into.clone())
}
+ (UInt8, Null | UInt8) => Some(type_into.clone()),
+ (UInt16, Null | UInt8 | UInt16) => Some(type_into.clone()),
+ (UInt32, Null | UInt8 | UInt16 | UInt32) => Some(type_into.clone()),
+ (UInt64, Null | UInt8 | UInt16 | UInt32 | UInt64) => Some(type_into.clone()),
+ (
+ Float32,
+ Null | Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64
+ | Float32,
+ ) => Some(type_into.clone()),
+ (
+ Float64,
+ Null
+ | Int8
+ | Int16
+ | Int32
+ | Int64
+ | UInt8
+ | UInt16
+ | UInt32
+ | UInt64
+ | Float32
+ | Float64
+ | Decimal128(_, _),
+ ) => Some(type_into.clone()),
+ (
+ Timestamp(TimeUnit::Nanosecond, None),
+ Null | Timestamp(_, None) | Date32 | Utf8 | LargeUtf8,
+ ) => Some(type_into.clone()),
+ (Interval(_), Utf8 | LargeUtf8) => Some(type_into.clone()),
// We can go into a Utf8View from a Utf8 or LargeUtf8
- (Utf8View, _) if matches!(type_from, Utf8 | LargeUtf8 | Null) => {
- Some(type_into.clone())
- }
+ (Utf8View, Utf8 | LargeUtf8 | Null) => Some(type_into.clone()),
// Any type can be coerced into strings
(Utf8 | LargeUtf8, _) => Some(type_into.clone()),
(Null, _) if can_cast_types(type_from, type_into) => Some(type_into.clone()),
- (List(_), _) if matches!(type_from, FixedSizeList(_, _)) => {
- Some(type_into.clone())
- }
+ (List(_), FixedSizeList(_, _)) => Some(type_into.clone()),
// Only accept list and largelist with the same number of dimensions unless the type is Null.
// List or LargeList with different dimensions should be handled in TypeSignature or other places before this
@@ -695,18 +724,16 @@ fn coerced_from<'a>(
Some(type_into.clone())
}
// should be able to coerce wildcard fixed size list to non wildcard fixed size list
- (FixedSizeList(f_into, FIXED_SIZE_LIST_WILDCARD), _) => match type_from {
- FixedSizeList(f_from, size_from) => {
- match coerced_from(f_into.data_type(), f_from.data_type()) {
- Some(data_type) if &data_type != f_into.data_type() => {
- let new_field =
- Arc::new(f_into.as_ref().clone().with_data_type(data_type));
- Some(FixedSizeList(new_field, *size_from))
- }
- Some(_) => Some(FixedSizeList(Arc::clone(f_into), *size_from)),
- _ => None,
- }
+ (
+ FixedSizeList(f_into, FIXED_SIZE_LIST_WILDCARD),
+ FixedSizeList(f_from, size_from),
+ ) => match coerced_from(f_into.data_type(), f_from.data_type()) {
+ Some(data_type) if &data_type != f_into.data_type() => {
+ let new_field =
+ Arc::new(f_into.as_ref().clone().with_data_type(data_type));
+ Some(FixedSizeList(new_field, *size_from))
}
+ Some(_) => Some(FixedSizeList(Arc::clone(f_into), *size_from)),
_ => None,
},
(Timestamp(unit, Some(tz)), _) if tz.as_ref() == TIMEZONE_WILDCARD => {
@@ -721,12 +748,7 @@ fn coerced_from<'a>(
_ => None,
}
}
- (Timestamp(_, Some(_)), _)
- if matches!(
- type_from,
- Null | Timestamp(_, _) | Date32 | Utf8 | LargeUtf8
- ) =>
- {
+ (Timestamp(_, Some(_)), Null | Timestamp(_, _) | Date32 | Utf8 | LargeUtf8) => {
Some(type_into.clone())
}
_ => None,
diff --git a/datafusion/expr/src/udaf.rs b/datafusion/expr/src/udaf.rs
index 780ea36910..6e48054bcf 100644
--- a/datafusion/expr/src/udaf.rs
+++ b/datafusion/expr/src/udaf.rs
@@ -36,8 +36,8 @@ use crate::function::{
use crate::groups_accumulator::GroupsAccumulator;
use crate::utils::format_state_name;
use crate::utils::AggregateOrderSensitivity;
-use crate::Signature;
use crate::{Accumulator, Expr};
+use crate::{Documentation, Signature};
/// Logical representation of a user-defined [aggregate function] (UDAF).
///
@@ -275,6 +275,14 @@ impl AggregateUDF {
pub fn default_value(&self, data_type: &DataType) -> Result<ScalarValue> {
self.inner.default_value(data_type)
}
+
+ /// Returns the documentation for this Aggregate UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ pub fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
}
impl<F> From<F> for AggregateUDF
@@ -299,25 +307,42 @@ where
/// # Basic Example
/// ```
/// # use std::any::Any;
+/// # use std::sync::OnceLock;
/// # use arrow::datatypes::DataType;
/// # use datafusion_common::{DataFusionError, plan_err, Result};
-/// # use datafusion_expr::{col, ColumnarValue, Signature, Volatility, Expr};
+/// # use datafusion_expr::{col, ColumnarValue, Signature, Volatility, Expr, Documentation};
/// # use datafusion_expr::{AggregateUDFImpl, AggregateUDF, Accumulator, function::{AccumulatorArgs, StateFieldsArgs}};
+/// # use datafusion_expr::window_doc_sections::DOC_SECTION_AGGREGATE;
/// # use arrow::datatypes::Schema;
/// # use arrow::datatypes::Field;
+///
/// #[derive(Debug, Clone)]
/// struct GeoMeanUdf {
-/// signature: Signature
+/// signature: Signature,
/// }
///
/// impl GeoMeanUdf {
/// fn new() -> Self {
/// Self {
-/// signature: Signature::uniform(1, vec![DataType::Float64], Volatility::Immutable)
+/// signature: Signature::uniform(1, vec![DataType::Float64], Volatility::Immutable),
/// }
/// }
/// }
///
+/// static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+///
+/// fn get_doc() -> &'static Documentation {
+/// DOCUMENTATION.get_or_init(|| {
+/// Documentation::builder()
+/// .with_doc_section(DOC_SECTION_AGGREGATE)
+/// .with_description("calculates a geometric mean")
+/// .with_syntax_example("geo_mean(2.0)")
+/// .with_argument("arg1", "The Float64 number for the geometric mean")
+/// .build()
+/// .unwrap()
+/// })
+/// }
+///
/// /// Implement the AggregateUDFImpl trait for GeoMeanUdf
/// impl AggregateUDFImpl for GeoMeanUdf {
/// fn as_any(&self) -> &dyn Any { self }
@@ -325,7 +350,7 @@ where
/// fn signature(&self) -> &Signature { &self.signature }
/// fn return_type(&self, args: &[DataType]) -> Result<DataType> {
/// if !matches!(args.get(0), Some(&DataType::Float64)) {
-/// return plan_err!("add_one only accepts Float64 arguments");
+/// return plan_err!("geo_mean only accepts Float64 arguments");
/// }
/// Ok(DataType::Float64)
/// }
@@ -337,6 +362,9 @@ where
/// Field::new("ordering", DataType::UInt32, true)
/// ])
/// }
+/// fn documentation(&self) -> Option<&Documentation> {
+/// Some(get_doc())
+/// }
/// }
///
/// // Create a new AggregateUDF from the implementation
@@ -603,6 +631,14 @@ pub trait AggregateUDFImpl: Debug + Send + Sync {
fn default_value(&self, data_type: &DataType) -> Result<ScalarValue> {
ScalarValue::try_from(data_type)
}
+
+ /// Returns the documentation for this Aggregate UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ fn documentation(&self) -> Option<&Documentation> {
+ None
+ }
}
impl PartialEq for dyn AggregateUDFImpl {
@@ -749,6 +785,41 @@ impl AggregateUDFImpl for AliasedAggregateUDFImpl {
fn is_descending(&self) -> Option<bool> {
self.inner.is_descending()
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
+}
+
+// Aggregate UDF doc sections for use in public documentation
+pub mod aggregate_doc_sections {
+ use crate::DocSection;
+
+ pub fn doc_sections() -> Vec<DocSection> {
+ vec![
+ DOC_SECTION_GENERAL,
+ DOC_SECTION_STATISTICAL,
+ DOC_SECTION_APPROXIMATE,
+ ]
+ }
+
+ pub const DOC_SECTION_GENERAL: DocSection = DocSection {
+ include: true,
+ label: "General Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_STATISTICAL: DocSection = DocSection {
+ include: true,
+ label: "Statistical Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_APPROXIMATE: DocSection = DocSection {
+ include: true,
+ label: "Approximate Functions",
+ description: None,
+ };
}
#[cfg(test)]
diff --git a/datafusion/expr/src/udf.rs b/datafusion/expr/src/udf.rs
index 938e1181d8..3759fb18f5 100644
--- a/datafusion/expr/src/udf.rs
+++ b/datafusion/expr/src/udf.rs
@@ -20,7 +20,9 @@
use crate::expr::schema_name_from_exprs_comma_seperated_without_space;
use crate::simplify::{ExprSimplifyResult, SimplifyInfo};
use crate::sort_properties::{ExprProperties, SortProperties};
-use crate::{ColumnarValue, Expr, ScalarFunctionImplementation, Signature};
+use crate::{
+ ColumnarValue, Documentation, Expr, ScalarFunctionImplementation, Signature,
+};
use arrow::datatypes::DataType;
use datafusion_common::{not_impl_err, ExprSchema, Result};
use datafusion_expr_common::interval_arithmetic::Interval;
@@ -274,6 +276,14 @@ impl ScalarUDF {
pub fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
self.inner.coerce_types(arg_types)
}
+
+ /// Returns the documentation for this Scalar UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ pub fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
}
impl<F> From<F> for ScalarUDF
@@ -298,22 +308,39 @@ where
/// # Basic Example
/// ```
/// # use std::any::Any;
+/// # use std::sync::OnceLock;
/// # use arrow::datatypes::DataType;
/// # use datafusion_common::{DataFusionError, plan_err, Result};
-/// # use datafusion_expr::{col, ColumnarValue, Signature, Volatility};
+/// # use datafusion_expr::{col, ColumnarValue, Documentation, Signature, Volatility};
/// # use datafusion_expr::{ScalarUDFImpl, ScalarUDF};
+/// # use datafusion_expr::scalar_doc_sections::DOC_SECTION_MATH;
+///
/// #[derive(Debug)]
/// struct AddOne {
-/// signature: Signature
+/// signature: Signature,
/// }
///
/// impl AddOne {
/// fn new() -> Self {
/// Self {
-/// signature: Signature::uniform(1, vec![DataType::Int32], Volatility::Immutable)
+/// signature: Signature::uniform(1, vec![DataType::Int32], Volatility::Immutable),
/// }
/// }
/// }
+///
+/// static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+///
+/// fn get_doc() -> &'static Documentation {
+/// DOCUMENTATION.get_or_init(|| {
+/// Documentation::builder()
+/// .with_doc_section(DOC_SECTION_MATH)
+/// .with_description("Add one to an int32")
+/// .with_syntax_example("add_one(2)")
+/// .with_argument("arg1", "The int32 number to add one to")
+/// .build()
+/// .unwrap()
+/// })
+/// }
///
/// /// Implement the ScalarUDFImpl trait for AddOne
/// impl ScalarUDFImpl for AddOne {
@@ -328,6 +355,9 @@ where
/// }
/// // The actual implementation would add one to the argument
/// fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> { unimplemented!() }
+/// fn documentation(&self) -> Option<&Documentation> {
+/// Some(get_doc())
+/// }
/// }
///
/// // Create a new ScalarUDF from the implementation
@@ -596,6 +626,14 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
self.signature().hash(hasher);
hasher.finish()
}
+
+ /// Returns the documentation for this Scalar UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ fn documentation(&self) -> Option<&Documentation> {
+ None
+ }
}
/// ScalarUDF that adds an alias to the underlying function. It is better to
@@ -709,4 +747,100 @@ impl ScalarUDFImpl for AliasedScalarUDFImpl {
self.aliases.hash(hasher);
hasher.finish()
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
+}
+
+// Scalar UDF doc sections for use in public documentation
+pub mod scalar_doc_sections {
+ use crate::DocSection;
+
+ pub fn doc_sections() -> Vec<DocSection> {
+ vec![
+ DOC_SECTION_MATH,
+ DOC_SECTION_CONDITIONAL,
+ DOC_SECTION_STRING,
+ DOC_SECTION_BINARY_STRING,
+ DOC_SECTION_REGEX,
+ DOC_SECTION_DATETIME,
+ DOC_SECTION_ARRAY,
+ DOC_SECTION_STRUCT,
+ DOC_SECTION_MAP,
+ DOC_SECTION_HASHING,
+ DOC_SECTION_OTHER,
+ ]
+ }
+
+ pub const DOC_SECTION_MATH: DocSection = DocSection {
+ include: true,
+ label: "Math Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_CONDITIONAL: DocSection = DocSection {
+ include: true,
+ label: "Conditional Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_STRING: DocSection = DocSection {
+ include: true,
+ label: "String Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_BINARY_STRING: DocSection = DocSection {
+ include: true,
+ label: "Binary String Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_REGEX: DocSection = DocSection {
+ include: true,
+ label: "Regular Expression Functions",
+ description: Some(
+ r#"Apache DataFusion uses a [PCRE-like](https://en.wikibooks.org/wiki/Regular_Expressions/Perl-Compatible_Regular_Expressions)
+regular expression [syntax](https://docs.rs/regex/latest/regex/#syntax)
+(minus support for several features including look-around and backreferences).
+The following regular expression functions are supported:"#,
+ ),
+ };
+
+ pub const DOC_SECTION_DATETIME: DocSection = DocSection {
+ include: true,
+ label: "Time and Date Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_ARRAY: DocSection = DocSection {
+ include: true,
+ label: "Array Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_STRUCT: DocSection = DocSection {
+ include: true,
+ label: "Struct Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_MAP: DocSection = DocSection {
+ include: true,
+ label: "Map Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_HASHING: DocSection = DocSection {
+ include: true,
+ label: "Hashing Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_OTHER: DocSection = DocSection {
+ include: true,
+ label: "Other Functions",
+ description: None,
+ };
}
diff --git a/datafusion/expr/src/udf_docs.rs b/datafusion/expr/src/udf_docs.rs
new file mode 100644
index 0000000000..e8245588d9
--- /dev/null
+++ b/datafusion/expr/src/udf_docs.rs
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::exec_err;
+use datafusion_common::Result;
+
+/// Documentation for use by [`ScalarUDFImpl`](crate::ScalarUDFImpl),
+/// [`AggregateUDFImpl`](crate::AggregateUDFImpl) and [`WindowUDFImpl`](crate::WindowUDFImpl) functions
+/// that will be used to generate public documentation.
+///
+/// The name of the udf will be pulled from the [`ScalarUDFImpl::name`](crate::ScalarUDFImpl::name),
+/// [`AggregateUDFImpl::name`](crate::AggregateUDFImpl::name) or [`WindowUDFImpl::name`](crate::WindowUDFImpl::name)
+/// function as appropriate.
+///
+/// All strings in the documentation are required to be
+/// in [markdown format](https://www.markdownguide.org/basic-syntax/).
+///
+/// Currently, documentation only supports a single language
+/// thus all text should be in English.
+#[derive(Debug, Clone)]
+pub struct Documentation {
+ /// the section in the documentation where the UDF will be documented
+ pub doc_section: DocSection,
+ /// the description for the UDF
+ pub description: String,
+ /// a brief example of the syntax. For example "ascii(str)"
+ pub syntax_example: String,
+ /// a sql example for the UDF, usually in the form of a sql prompt
+ /// query and output. It is strongly recommended to provide an
+ /// example for anything but the most basic UDF's
+ pub sql_example: Option<String>,
+ /// arguments for the UDF which will be displayed in array order.
+ /// Left member of a pair is the argument name, right is a
+ /// description for the argument
+ pub arguments: Option<Vec<(String, String)>>,
+ /// related functions if any. Values should match the related
+ /// udf's name exactly. Related udf's must be of the same
+ /// UDF type (scalar, aggregate or window) for proper linking to
+ /// occur
+ pub related_udfs: Option<Vec<String>>,
+}
+
+impl Documentation {
+ /// Returns a new [`DocumentationBuilder`] with no options set.
+ pub fn builder() -> DocumentationBuilder {
+ DocumentationBuilder::new()
+ }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct DocSection {
+ /// true to include this doc section in the public
+ /// documentation, false otherwise
+ pub include: bool,
+ /// a display label for the doc section. For example: "Math Expressions"
+ pub label: &'static str,
+ /// an optional description for the doc section
+ pub description: Option<&'static str>,
+}
+
+/// A builder to be used for building [`Documentation`]'s.
+///
+/// Example:
+///
+/// ```rust
+/// # use datafusion_expr::Documentation;
+/// # use datafusion_expr::scalar_doc_sections::DOC_SECTION_MATH;
+/// # use datafusion_common::Result;
+/// #
+/// # fn main() -> Result<()> {
+/// let documentation = Documentation::builder()
+/// .with_doc_section(DOC_SECTION_MATH)
+/// .with_description("Add one to an int32")
+/// .with_syntax_example("add_one(2)")
+/// .with_argument("arg_1", "The int32 number to add one to")
+/// .build()?;
+/// Ok(())
+/// # }
+pub struct DocumentationBuilder {
+ pub doc_section: Option<DocSection>,
+ pub description: Option<String>,
+ pub syntax_example: Option<String>,
+ pub sql_example: Option<String>,
+ pub arguments: Option<Vec<(String, String)>>,
+ pub related_udfs: Option<Vec<String>>,
+}
+
+impl DocumentationBuilder {
+ pub fn new() -> Self {
+ Self {
+ doc_section: None,
+ description: None,
+ syntax_example: None,
+ sql_example: None,
+ arguments: None,
+ related_udfs: None,
+ }
+ }
+
+ pub fn with_doc_section(mut self, doc_section: DocSection) -> Self {
+ self.doc_section = Some(doc_section);
+ self
+ }
+
+ pub fn with_description(mut self, description: impl Into<String>) -> Self {
+ self.description = Some(description.into());
+ self
+ }
+
+ pub fn with_syntax_example(mut self, syntax_example: impl Into<String>) -> Self {
+ self.syntax_example = Some(syntax_example.into());
+ self
+ }
+
+ pub fn with_sql_example(mut self, sql_example: impl Into<String>) -> Self {
+ self.sql_example = Some(sql_example.into());
+ self
+ }
+
+ /// Adds documentation for a specific argument to the documentation.
+ ///
+ /// Arguments are displayed in the order they are added.
+ pub fn with_argument(
+ mut self,
+ arg_name: impl Into<String>,
+ arg_description: impl Into<String>,
+ ) -> Self {
+ let mut args = self.arguments.unwrap_or_default();
+ args.push((arg_name.into(), arg_description.into()));
+ self.arguments = Some(args);
+ self
+ }
+
+ /// Add a standard "expression" argument to the documentation
+ ///
+ /// This is similar to [`Self::with_argument`] except that a standard
+ /// description is appended to the end: `"Can be a constant, column, or
+ /// function, and any combination of arithmetic operators."`
+ ///
+ /// The argument is rendered like
+ ///
+ /// ```text
+ /// <arg_name>:
+ /// <expression_type> expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators.
+ /// ```
+ pub fn with_standard_argument(
+ self,
+ arg_name: impl Into<String>,
+ expression_type: impl AsRef<str>,
+ ) -> Self {
+ let expression_type = expression_type.as_ref();
+ self.with_argument(arg_name, format!("{expression_type} expression to operate on. Can be a constant, column, or function, and any combination of operators."))
+ }
+
+ pub fn with_related_udf(mut self, related_udf: impl Into<String>) -> Self {
+ let mut related = self.related_udfs.unwrap_or_default();
+ related.push(related_udf.into());
+ self.related_udfs = Some(related);
+ self
+ }
+
+ pub fn build(self) -> Result<Documentation> {
+ let Self {
+ doc_section,
+ description,
+ syntax_example,
+ sql_example,
+ arguments,
+ related_udfs,
+ } = self;
+
+ if doc_section.is_none() {
+ return exec_err!("Documentation must have a doc section");
+ }
+ if description.is_none() {
+ return exec_err!("Documentation must have a description");
+ }
+ if syntax_example.is_none() {
+ return exec_err!("Documentation must have a syntax_example");
+ }
+
+ Ok(Documentation {
+ doc_section: doc_section.unwrap(),
+ description: description.unwrap(),
+ syntax_example: syntax_example.unwrap(),
+ sql_example,
+ arguments,
+ related_udfs,
+ })
+ }
+}
+
+impl Default for DocumentationBuilder {
+ fn default() -> Self {
+ Self::new()
+ }
+}
diff --git a/datafusion/expr/src/udwf.rs b/datafusion/expr/src/udwf.rs
index 678a0b62cd..69f357d48f 100644
--- a/datafusion/expr/src/udwf.rs
+++ b/datafusion/expr/src/udwf.rs
@@ -33,7 +33,8 @@ use datafusion_functions_window_common::field::WindowUDFFieldArgs;
use crate::expr::WindowFunction;
use crate::{
- function::WindowFunctionSimplification, Expr, PartitionEvaluator, Signature,
+ function::WindowFunctionSimplification, Documentation, Expr, PartitionEvaluator,
+ Signature,
};
/// Logical representation of a user-defined window function (UDWF)
@@ -180,6 +181,14 @@ impl WindowUDF {
pub fn reverse_expr(&self) -> ReversedUDWF {
self.inner.reverse_expr()
}
+
+ /// Returns the documentation for this Window UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ pub fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
}
impl<F> From<F> for WindowUDF
@@ -204,30 +213,47 @@ where
/// # Basic Example
/// ```
/// # use std::any::Any;
+/// # use std::sync::OnceLock;
/// # use arrow::datatypes::{DataType, Field};
/// # use datafusion_common::{DataFusionError, plan_err, Result};
-/// # use datafusion_expr::{col, Signature, Volatility, PartitionEvaluator, WindowFrame, ExprFunctionExt};
+/// # use datafusion_expr::{col, Signature, Volatility, PartitionEvaluator, WindowFrame, ExprFunctionExt, Documentation};
/// # use datafusion_expr::{WindowUDFImpl, WindowUDF};
-/// use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// # use datafusion_expr::window_doc_sections::DOC_SECTION_ANALYTICAL;
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+///
/// #[derive(Debug, Clone)]
/// struct SmoothIt {
-/// signature: Signature
+/// signature: Signature,
/// }
///
/// impl SmoothIt {
/// fn new() -> Self {
/// Self {
-/// signature: Signature::uniform(1, vec![DataType::Int32], Volatility::Immutable)
+/// signature: Signature::uniform(1, vec![DataType::Int32], Volatility::Immutable),
/// }
/// }
/// }
///
-/// /// Implement the WindowUDFImpl trait for AddOne
+/// static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+///
+/// fn get_doc() -> &'static Documentation {
+/// DOCUMENTATION.get_or_init(|| {
+/// Documentation::builder()
+/// .with_doc_section(DOC_SECTION_ANALYTICAL)
+/// .with_description("smooths the windows")
+/// .with_syntax_example("smooth_it(2)")
+/// .with_argument("arg1", "The int32 number to smooth by")
+/// .build()
+/// .unwrap()
+/// })
+/// }
+///
+/// /// Implement the WindowUDFImpl trait for SmoothIt
/// impl WindowUDFImpl for SmoothIt {
/// fn as_any(&self) -> &dyn Any { self }
/// fn name(&self) -> &str { "smooth_it" }
/// fn signature(&self) -> &Signature { &self.signature }
-/// // The actual implementation would add one to the argument
+/// // The actual implementation would smooth the window
/// fn partition_evaluator(&self) -> Result<Box<dyn PartitionEvaluator>> { unimplemented!() }
/// fn field(&self, field_args: WindowUDFFieldArgs) -> Result<Field> {
/// if let Some(DataType::Int32) = field_args.get_input_type(0) {
@@ -236,6 +262,9 @@ where
/// plan_err!("smooth_it only accepts Int32 arguments")
/// }
/// }
+/// fn documentation(&self) -> Option<&Documentation> {
+/// Some(get_doc())
+/// }
/// }
///
/// // Create a new WindowUDF from the implementation
@@ -326,6 +355,10 @@ pub trait WindowUDFImpl: Debug + Send + Sync {
}
/// The [`Field`] of the final result of evaluating this window function.
+ ///
+ /// Call `field_args.name()` to get the fully qualified name for defining
+ /// the [`Field`]. For a complete example see the implementation in the
+ /// [Basic Example](WindowUDFImpl#basic-example) section.
fn field(&self, field_args: WindowUDFFieldArgs) -> Result<Field>;
/// Allows the window UDF to define a custom result ordering.
@@ -365,6 +398,14 @@ pub trait WindowUDFImpl: Debug + Send + Sync {
fn reverse_expr(&self) -> ReversedUDWF {
ReversedUDWF::NotSupported
}
+
+ /// Returns the documentation for this Window UDF.
+ ///
+ /// Documentation can be accessed programmatically as well as
+ /// generating publicly facing documentation.
+ fn documentation(&self) -> Option<&Documentation> {
+ None
+ }
}
pub enum ReversedUDWF {
@@ -465,6 +506,41 @@ impl WindowUDFImpl for AliasedWindowUDFImpl {
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
self.inner.coerce_types(arg_types)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.inner.documentation()
+ }
+}
+
+// Window UDF doc sections for use in public documentation
+pub mod window_doc_sections {
+ use crate::DocSection;
+
+ pub fn doc_sections() -> Vec<DocSection> {
+ vec![
+ DOC_SECTION_AGGREGATE,
+ DOC_SECTION_RANKING,
+ DOC_SECTION_ANALYTICAL,
+ ]
+ }
+
+ pub const DOC_SECTION_AGGREGATE: DocSection = DocSection {
+ include: true,
+ label: "Aggregate Functions",
+ description: Some("All aggregate functions can be used as window functions."),
+ };
+
+ pub const DOC_SECTION_RANKING: DocSection = DocSection {
+ include: true,
+ label: "Ranking Functions",
+ description: None,
+ };
+
+ pub const DOC_SECTION_ANALYTICAL: DocSection = DocSection {
+ include: true,
+ label: "Analytical Functions",
+ description: None,
+ };
}
#[cfg(test)]
diff --git a/datafusion/expr/src/utils.rs b/datafusion/expr/src/utils.rs
index 1d8eb9445e..02b36d0fea 100644
--- a/datafusion/expr/src/utils.rs
+++ b/datafusion/expr/src/utils.rs
@@ -19,6 +19,7 @@
use std::cmp::Ordering;
use std::collections::{HashMap, HashSet};
+use std::ops::Deref;
use std::sync::Arc;
use crate::expr::{Alias, Sort, WildcardOptions, WindowFunction};
@@ -38,6 +39,7 @@ use datafusion_common::{
DataFusionError, Result, TableReference,
};
+use indexmap::IndexSet;
use sqlparser::ast::{ExceptSelectItem, ExcludeSelectItem};
pub use datafusion_functions_aggregate_common::order::AggregateOrderSensitivity;
@@ -65,9 +67,10 @@ pub fn grouping_set_expr_count(group_expr: &[Expr]) -> Result<usize> {
"Invalid group by expressions, GroupingSet must be the only expression"
);
}
- Ok(grouping_set.distinct_expr().len())
+ // Groupings sets have an additional interal column for the grouping id
+ Ok(grouping_set.distinct_expr().len() + 1)
} else {
- Ok(group_expr.len())
+ grouping_set_to_exprlist(group_expr).map(|exprs| exprs.len())
}
}
@@ -260,7 +263,11 @@ pub fn grouping_set_to_exprlist(group_expr: &[Expr]) -> Result<Vec<&Expr>> {
}
Ok(grouping_set.distinct_expr())
} else {
- Ok(group_expr.iter().collect())
+ Ok(group_expr
+ .iter()
+ .collect::<IndexSet<_>>()
+ .into_iter()
+ .collect())
}
}
@@ -758,6 +765,15 @@ pub fn find_base_plan(input: &LogicalPlan) -> &LogicalPlan {
match input {
LogicalPlan::Window(window) => find_base_plan(&window.input),
LogicalPlan::Aggregate(agg) => find_base_plan(&agg.input),
+ // [SqlToRel::try_process_unnest] will convert Expr(Unnest(Expr)) to Projection/Unnest/Projection
+ // We should expand the wildcard expression based on the input plan of the inner Projection.
+ LogicalPlan::Unnest(unnest) => {
+ if let LogicalPlan::Projection(projection) = unnest.input.deref() {
+ find_base_plan(&projection.input)
+ } else {
+ input
+ }
+ }
LogicalPlan::Filter(filter) => {
if filter.having {
// If a filter is used for a having clause, its input plan is an aggregation.
diff --git a/datafusion/functions-aggregate/Cargo.toml b/datafusion/functions-aggregate/Cargo.toml
index d78f68a260..37e4c7f4a5 100644
--- a/datafusion/functions-aggregate/Cargo.toml
+++ b/datafusion/functions-aggregate/Cargo.toml
@@ -48,9 +48,9 @@ datafusion-functions-aggregate-common = { workspace = true }
datafusion-physical-expr = { workspace = true }
datafusion-physical-expr-common = { workspace = true }
half = { workspace = true }
+indexmap = { workspace = true }
log = { workspace = true }
paste = "1.0.14"
-sqlparser = { workspace = true }
[dev-dependencies]
arrow = { workspace = true, features = ["test_utils"] }
diff --git a/datafusion/functions-aggregate/src/bit_and_or_xor.rs b/datafusion/functions-aggregate/src/bit_and_or_xor.rs
index aa65062e33..c5382c168f 100644
--- a/datafusion/functions-aggregate/src/bit_and_or_xor.rs
+++ b/datafusion/functions-aggregate/src/bit_and_or_xor.rs
@@ -35,11 +35,14 @@ use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
use datafusion_expr::type_coercion::aggregates::INTEGERS;
use datafusion_expr::utils::format_state_name;
use datafusion_expr::{
- Accumulator, AggregateUDFImpl, GroupsAccumulator, ReversedUDAF, Signature, Volatility,
+ Accumulator, AggregateUDFImpl, Documentation, GroupsAccumulator, ReversedUDAF,
+ Signature, Volatility,
};
+use datafusion_expr::aggregate_doc_sections::DOC_SECTION_GENERAL;
use datafusion_functions_aggregate_common::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
use std::ops::{BitAndAssign, BitOrAssign, BitXorAssign};
+use std::sync::OnceLock;
/// This macro helps create group accumulators based on bitwise operations typically used internally
/// and might not be necessary for users to call directly.
@@ -110,8 +113,9 @@ macro_rules! downcast_bitwise_accumulator {
/// `EXPR_FN` identifier used to name the generated expression function.
/// `AGGREGATE_UDF_FN` is an identifier used to name the underlying UDAF function.
/// `OPR_TYPE` is an expression that evaluates to the type of bitwise operation to be performed.
+/// `DOCUMENTATION` documentation for the UDAF
macro_rules! make_bitwise_udaf_expr_and_func {
- ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr) => {
+ ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr, $DOCUMENTATION:expr) => {
make_udaf_expr!(
$EXPR_FN,
expr_x,
@@ -125,14 +129,73 @@ macro_rules! make_bitwise_udaf_expr_and_func {
create_func!(
$EXPR_FN,
$AGGREGATE_UDF_FN,
- BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN))
+ BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN), $DOCUMENTATION)
);
};
}
-make_bitwise_udaf_expr_and_func!(bit_and, bit_and_udaf, BitwiseOperationType::And);
-make_bitwise_udaf_expr_and_func!(bit_or, bit_or_udaf, BitwiseOperationType::Or);
-make_bitwise_udaf_expr_and_func!(bit_xor, bit_xor_udaf, BitwiseOperationType::Xor);
+static BIT_AND_DOC: OnceLock<Documentation> = OnceLock::new();
+
+fn get_bit_and_doc() -> &'static Documentation {
+ BIT_AND_DOC.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_GENERAL)
+ .with_description("Computes the bitwise AND of all non-null input values.")
+ .with_syntax_example("bit_and(expression)")
+ .with_standard_argument("expression", "Integer")
+ .build()
+ .unwrap()
+ })
+}
+
+static BIT_OR_DOC: OnceLock<Documentation> = OnceLock::new();
+
+fn get_bit_or_doc() -> &'static Documentation {
+ BIT_OR_DOC.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_GENERAL)
+ .with_description("Computes the bitwise OR of all non-null input values.")
+ .with_syntax_example("bit_or(expression)")
+ .with_standard_argument("expression", "Integer")
+ .build()
+ .unwrap()
+ })
+}
+
+static BIT_XOR_DOC: OnceLock<Documentation> = OnceLock::new();
+
+fn get_bit_xor_doc() -> &'static Documentation {
+ BIT_XOR_DOC.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_GENERAL)
+ .with_description(
+ "Computes the bitwise exclusive OR of all non-null input values.",
+ )
+ .with_syntax_example("bit_xor(expression)")
+ .with_standard_argument("expression", "Integer")
+ .build()
+ .unwrap()
+ })
+}
+
+make_bitwise_udaf_expr_and_func!(
+ bit_and,
+ bit_and_udaf,
+ BitwiseOperationType::And,
+ get_bit_and_doc()
+);
+make_bitwise_udaf_expr_and_func!(
+ bit_or,
+ bit_or_udaf,
+ BitwiseOperationType::Or,
+ get_bit_or_doc()
+);
+make_bitwise_udaf_expr_and_func!(
+ bit_xor,
+ bit_xor_udaf,
+ BitwiseOperationType::Xor,
+ get_bit_xor_doc()
+);
/// The different types of bitwise operations that can be performed.
#[derive(Debug, Clone, Eq, PartialEq)]
@@ -155,14 +218,20 @@ struct BitwiseOperation {
/// `operation` indicates the type of bitwise operation to be performed.
operation: BitwiseOperationType,
func_name: &'static str,
+ documentation: &'static Documentation,
}
impl BitwiseOperation {
- pub fn new(operator: BitwiseOperationType, func_name: &'static str) -> Self {
+ pub fn new(
+ operator: BitwiseOperationType,
+ func_name: &'static str,
+ documentation: &'static Documentation,
+ ) -> Self {
Self {
operation: operator,
signature: Signature::uniform(1, INTEGERS.to_vec(), Volatility::Immutable),
func_name,
+ documentation,
}
}
}
@@ -239,6 +308,10 @@ impl AggregateUDFImpl for BitwiseOperation {
fn reverse_expr(&self) -> ReversedUDAF {
ReversedUDAF::Identical
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(self.documentation)
+ }
}
struct BitAndAccumulator<T: ArrowNumericType> {
diff --git a/datafusion/functions-aggregate/src/macros.rs b/datafusion/functions-aggregate/src/macros.rs
index 573b9fd5bd..ffb5183278 100644
--- a/datafusion/functions-aggregate/src/macros.rs
+++ b/datafusion/functions-aggregate/src/macros.rs
@@ -15,23 +15,6 @@
// specific language governing permissions and limitations
// under the License.
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
macro_rules! make_udaf_expr {
($EXPR_FN:ident, $($arg:ident)*, $DOC:expr, $AGGREGATE_UDF_FN:ident) => {
// "fluent expr_fn" style function
diff --git a/datafusion/functions-aggregate/src/min_max.rs b/datafusion/functions-aggregate/src/min_max.rs
index 1ce1abe09e..e0b029f090 100644
--- a/datafusion/functions-aggregate/src/min_max.rs
+++ b/datafusion/functions-aggregate/src/min_max.rs
@@ -1240,26 +1240,24 @@ impl Accumulator for SlidingMinAccumulator {
}
}
-//
-// Moving min and moving max
-// The implementation is taken from https://github.com/spebern/moving_min_max/blob/master/src/lib.rs.
-
-// Keep track of the minimum or maximum value in a sliding window.
-//
-// `moving min max` provides one data structure for keeping track of the
-// minimum value and one for keeping track of the maximum value in a sliding
-// window.
-//
-// Each element is stored with the current min/max. One stack to push and another one for pop. If pop stack is empty,
-// push to this stack all elements popped from first stack while updating their current min/max. Now pop from
-// the second stack (MovingMin/Max struct works as a queue). To find the minimum element of the queue,
-// look at the smallest/largest two elements of the individual stacks, then take the minimum of those two values.
-//
-// The complexity of the operations are
-// - O(1) for getting the minimum/maximum
-// - O(1) for push
-// - amortized O(1) for pop
-
+/// Keep track of the minimum value in a sliding window.
+///
+/// The implementation is taken from <https://github.com/spebern/moving_min_max/blob/master/src/lib.rs>
+///
+/// `moving min max` provides one data structure for keeping track of the
+/// minimum value and one for keeping track of the maximum value in a sliding
+/// window.
+///
+/// Each element is stored with the current min/max. One stack to push and another one for pop. If pop stack is empty,
+/// push to this stack all elements popped from first stack while updating their current min/max. Now pop from
+/// the second stack (MovingMin/Max struct works as a queue). To find the minimum element of the queue,
+/// look at the smallest/largest two elements of the individual stacks, then take the minimum of those two values.
+///
+/// The complexity of the operations are
+/// - O(1) for getting the minimum/maximum
+/// - O(1) for push
+/// - amortized O(1) for pop
+///
/// ```
/// # use datafusion_functions_aggregate::min_max::MovingMin;
/// let mut moving_min = MovingMin::<i32>::new();
@@ -1375,6 +1373,11 @@ impl<T: Clone + PartialOrd> MovingMin<T> {
self.len() == 0
}
}
+
+/// Keep track of the maximum value in a sliding window.
+///
+/// See [`MovingMin`] for more details.
+///
/// ```
/// # use datafusion_functions_aggregate::min_max::MovingMax;
/// let mut moving_max = MovingMax::<i32>::new();
diff --git a/datafusion/functions-aggregate/src/variance.rs b/datafusion/functions-aggregate/src/variance.rs
index 3648ec0d13..49a30344c2 100644
--- a/datafusion/functions-aggregate/src/variance.rs
+++ b/datafusion/functions-aggregate/src/variance.rs
@@ -18,22 +18,24 @@
//! [`VarianceSample`]: variance sample aggregations.
//! [`VariancePopulation`]: variance population aggregations.
-use std::{fmt::Debug, sync::Arc};
-
use arrow::{
array::{Array, ArrayRef, BooleanArray, Float64Array, UInt64Array},
buffer::NullBuffer,
compute::kernels::cast,
datatypes::{DataType, Field},
};
+use std::sync::OnceLock;
+use std::{fmt::Debug, sync::Arc};
use datafusion_common::{
downcast_value, not_impl_err, plan_err, DataFusionError, Result, ScalarValue,
};
+use datafusion_expr::aggregate_doc_sections::DOC_SECTION_GENERAL;
use datafusion_expr::{
function::{AccumulatorArgs, StateFieldsArgs},
utils::format_state_name,
- Accumulator, AggregateUDFImpl, GroupsAccumulator, Signature, Volatility,
+ Accumulator, AggregateUDFImpl, Documentation, GroupsAccumulator, Signature,
+ Volatility,
};
use datafusion_functions_aggregate_common::{
aggregate::groups_accumulator::accumulate::accumulate, stats::StatsType,
@@ -135,6 +137,26 @@ impl AggregateUDFImpl for VarianceSample {
) -> Result<Box<dyn GroupsAccumulator>> {
Ok(Box::new(VarianceGroupsAccumulator::new(StatsType::Sample)))
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_variance_sample_doc())
+ }
+}
+
+static VARIANCE_SAMPLE_DOC: OnceLock<Documentation> = OnceLock::new();
+
+fn get_variance_sample_doc() -> &'static Documentation {
+ VARIANCE_SAMPLE_DOC.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_GENERAL)
+ .with_description(
+ "Returns the statistical sample variance of a set of numbers.",
+ )
+ .with_syntax_example("var(expression)")
+ .with_standard_argument("expression", "Numeric")
+ .build()
+ .unwrap()
+ })
}
pub struct VariancePopulation {
@@ -222,6 +244,25 @@ impl AggregateUDFImpl for VariancePopulation {
StatsType::Population,
)))
}
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_variance_population_doc())
+ }
+}
+
+static VARIANCE_POPULATION_DOC: OnceLock<Documentation> = OnceLock::new();
+
+fn get_variance_population_doc() -> &'static Documentation {
+ VARIANCE_POPULATION_DOC.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_GENERAL)
+ .with_description(
+ "Returns the statistical population variance of a set of numbers.",
+ )
+ .with_syntax_example("var_pop(expression)")
+ .with_standard_argument("expression", "Numeric")
+ .build()
+ .unwrap()
+ })
}
/// An accumulator to compute variance
diff --git a/datafusion/functions-nested/src/make_array.rs b/datafusion/functions-nested/src/make_array.rs
index 79858041d3..51fc71e6b0 100644
--- a/datafusion/functions-nested/src/make_array.rs
+++ b/datafusion/functions-nested/src/make_array.rs
@@ -17,6 +17,7 @@
//! [`ScalarUDFImpl`] definitions for `make_array` function.
+use std::vec;
use std::{any::Any, sync::Arc};
use arrow::array::{ArrayData, Capacities, MutableArrayData};
@@ -26,9 +27,8 @@ use arrow_array::{
use arrow_buffer::OffsetBuffer;
use arrow_schema::DataType::{LargeList, List, Null};
use arrow_schema::{DataType, Field};
-use datafusion_common::internal_err;
use datafusion_common::{plan_err, utils::array_into_list_array_nullable, Result};
-use datafusion_expr::type_coercion::binary::comparison_coercion;
+use datafusion_expr::binary::type_union_resolution;
use datafusion_expr::TypeSignature;
use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
@@ -82,19 +82,12 @@ impl ScalarUDFImpl for MakeArray {
match arg_types.len() {
0 => Ok(empty_array_type()),
_ => {
- let mut expr_type = DataType::Null;
- for arg_type in arg_types {
- if !arg_type.equals_datatype(&DataType::Null) {
- expr_type = arg_type.clone();
- break;
- }
- }
-
- if expr_type.is_null() {
- expr_type = DataType::Int64;
- }
-
- Ok(List(Arc::new(Field::new("item", expr_type, true))))
+ // At this point, all the type in array should be coerced to the same one
+ Ok(List(Arc::new(Field::new(
+ "item",
+ arg_types[0].to_owned(),
+ true,
+ ))))
}
}
}
@@ -112,22 +105,21 @@ impl ScalarUDFImpl for MakeArray {
}
fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
- let new_type = arg_types.iter().skip(1).try_fold(
- arg_types.first().unwrap().clone(),
- |acc, x| {
- // The coerced types found by `comparison_coercion` are not guaranteed to be
- // coercible for the arguments. `comparison_coercion` returns more loose
- // types that can be coerced to both `acc` and `x` for comparison purpose.
- // See `maybe_data_types` for the actual coercion.
- let coerced_type = comparison_coercion(&acc, x);
- if let Some(coerced_type) = coerced_type {
- Ok(coerced_type)
- } else {
- internal_err!("Coercion from {acc:?} to {x:?} failed.")
- }
- },
- )?;
- Ok(vec![new_type; arg_types.len()])
+ if let Some(new_type) = type_union_resolution(arg_types) {
+ if let DataType::FixedSizeList(field, _) = new_type {
+ Ok(vec![DataType::List(field); arg_types.len()])
+ } else if new_type.is_null() {
+ Ok(vec![DataType::Int64; arg_types.len()])
+ } else {
+ Ok(vec![new_type; arg_types.len()])
+ }
+ } else {
+ plan_err!(
+ "Fail to find the valid type between {:?} for {}",
+ arg_types,
+ self.name()
+ )
+ }
}
}
diff --git a/datafusion/functions-window/Cargo.toml b/datafusion/functions-window/Cargo.toml
index 8dcec6bc96..952e5720c7 100644
--- a/datafusion/functions-window/Cargo.toml
+++ b/datafusion/functions-window/Cargo.toml
@@ -43,6 +43,7 @@ datafusion-expr = { workspace = true }
datafusion-functions-window-common = { workspace = true }
datafusion-physical-expr-common = { workspace = true }
log = { workspace = true }
+paste = "1.0.15"
[dev-dependencies]
arrow = { workspace = true }
diff --git a/datafusion/functions-window/src/lib.rs b/datafusion/functions-window/src/lib.rs
index 790a500f1f..6e98bb0914 100644
--- a/datafusion/functions-window/src/lib.rs
+++ b/datafusion/functions-window/src/lib.rs
@@ -29,6 +29,8 @@ use log::debug;
use datafusion_expr::registry::FunctionRegistry;
use datafusion_expr::WindowUDF;
+#[macro_use]
+pub mod macros;
pub mod row_number;
/// Fluent-style API for creating `Expr`s
diff --git a/datafusion/functions-window/src/macros.rs b/datafusion/functions-window/src/macros.rs
new file mode 100644
index 0000000000..843d8ecb38
--- /dev/null
+++ b/datafusion/functions-window/src/macros.rs
@@ -0,0 +1,674 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Convenience macros for defining a user-defined window function
+//! and associated expression API (fluent style).
+//!
+//! See [`define_udwf_and_expr!`] for usage examples.
+//!
+//! [`define_udwf_and_expr!`]: crate::define_udwf_and_expr!
+
+/// Lazily initializes a user-defined window function exactly once
+/// when called concurrently. Repeated calls return a reference to the
+/// same instance.
+///
+/// # Parameters
+///
+/// * `$UDWF`: The struct which defines the [`Signature`](datafusion_expr::Signature)
+/// of the user-defined window function.
+/// * `$OUT_FN_NAME`: The basename to generate a unique function name like
+/// `$OUT_FN_NAME_udwf`.
+/// * `$DOC`: Doc comments for UDWF.
+/// * (optional) `$CTOR`: Pass a custom constructor. When omitted it
+/// automatically resolves to `$UDWF::default()`.
+///
+/// # Example
+///
+/// ```
+/// # use std::any::Any;
+/// # use datafusion_common::arrow::datatypes::{DataType, Field};
+/// # use datafusion_expr::{PartitionEvaluator, Signature, Volatility, WindowUDFImpl};
+/// #
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// # use datafusion_functions_window::get_or_init_udwf;
+/// #
+/// /// Defines the `simple_udwf()` user-defined window function.
+/// get_or_init_udwf!(
+/// SimpleUDWF,
+/// simple,
+/// "Simple user-defined window function doc comment."
+/// );
+/// #
+/// # assert_eq!(simple_udwf().name(), "simple_user_defined_window_function");
+/// #
+/// # #[derive(Debug)]
+/// # struct SimpleUDWF {
+/// # signature: Signature,
+/// # }
+/// #
+/// # impl Default for SimpleUDWF {
+/// # fn default() -> Self {
+/// # Self {
+/// # signature: Signature::any(0, Volatility::Immutable),
+/// # }
+/// # }
+/// # }
+/// #
+/// # impl WindowUDFImpl for SimpleUDWF {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "simple_user_defined_window_function"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(field_args.name(), DataType::Int64, false))
+/// # }
+/// # }
+/// #
+/// ```
+#[macro_export]
+macro_rules! get_or_init_udwf {
+ ($UDWF:ident, $OUT_FN_NAME:ident, $DOC:expr) => {
+ get_or_init_udwf!($UDWF, $OUT_FN_NAME, $DOC, $UDWF::default);
+ };
+
+ ($UDWF:ident, $OUT_FN_NAME:ident, $DOC:expr, $CTOR:path) => {
+ paste::paste! {
+ #[doc = concat!(" Singleton instance of [`", stringify!($OUT_FN_NAME), "`], ensures the user-defined")]
+ #[doc = concat!(" window function is only created once.")]
+ #[allow(non_upper_case_globals)]
+ static [<STATIC_ $UDWF>]: std::sync::OnceLock<std::sync::Arc<datafusion_expr::WindowUDF>> =
+ std::sync::OnceLock::new();
+
+ #[doc = concat!(" Returns a [`WindowUDF`](datafusion_expr::WindowUDF) for [`", stringify!($OUT_FN_NAME), "`].")]
+ #[doc = ""]
+ #[doc = concat!(" ", $DOC)]
+ pub fn [<$OUT_FN_NAME _udwf>]() -> std::sync::Arc<datafusion_expr::WindowUDF> {
+ [<STATIC_ $UDWF>]
+ .get_or_init(|| {
+ std::sync::Arc::new(datafusion_expr::WindowUDF::from($CTOR()))
+ })
+ .clone()
+ }
+ }
+ };
+}
+
+/// Create a [`WindowFunction`] expression that exposes a fluent API
+/// which you can use to build more complex expressions.
+///
+/// [`WindowFunction`]: datafusion_expr::Expr::WindowFunction
+///
+/// # Parameters
+///
+/// * `$UDWF`: The struct which defines the [`Signature`] of the
+/// user-defined window function.
+/// * `$OUT_FN_NAME`: The basename to generate a unique function name like
+/// `$OUT_FN_NAME_udwf`.
+/// * `$DOC`: Doc comments for UDWF.
+/// * (optional) `[$($PARAM:ident),+]`: An array of 1 or more parameters
+/// for the generated function. The type of parameters is [`Expr`].
+/// When omitted this creates a function with zero parameters.
+///
+/// [`Signature`]: datafusion_expr::Signature
+/// [`Expr`]: datafusion_expr::Expr
+///
+/// # Example
+///
+/// 1. With Zero Parameters
+/// ```
+/// # use std::any::Any;
+/// # use datafusion_common::arrow::datatypes::{DataType, Field};
+/// # use datafusion_expr::{PartitionEvaluator, Signature, Volatility, WindowUDFImpl};
+/// # use datafusion_functions_window::{create_udwf_expr, get_or_init_udwf};
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// # get_or_init_udwf!(
+/// # RowNumber,
+/// # row_number,
+/// # "Returns a unique row number for each row in window partition beginning at 1."
+/// # );
+/// /// Creates `row_number()` API which has zero parameters:
+/// ///
+/// /// ```
+/// /// /// Returns a unique row number for each row in window partition
+/// /// /// beginning at 1.
+/// /// pub fn row_number() -> datafusion_expr::Expr {
+/// /// row_number_udwf().call(vec![])
+/// /// }
+/// /// ```
+/// create_udwf_expr!(
+/// RowNumber,
+/// row_number,
+/// "Returns a unique row number for each row in window partition beginning at 1."
+/// );
+/// #
+/// # assert_eq!(
+/// # row_number().name_for_alias().unwrap(),
+/// # "row_number() ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
+/// # );
+/// #
+/// # #[derive(Debug)]
+/// # struct RowNumber {
+/// # signature: Signature,
+/// # }
+/// # impl Default for RowNumber {
+/// # fn default() -> Self {
+/// # Self {
+/// # signature: Signature::any(0, Volatility::Immutable),
+/// # }
+/// # }
+/// # }
+/// # impl WindowUDFImpl for RowNumber {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "row_number"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(field_args.name(), DataType::UInt64, false))
+/// # }
+/// # }
+/// ```
+///
+/// 2. With Multiple Parameters
+/// ```
+/// # use std::any::Any;
+/// #
+/// # use datafusion_expr::{
+/// # PartitionEvaluator, Signature, TypeSignature, Volatility, WindowUDFImpl,
+/// # };
+/// #
+/// # use datafusion_functions_window::{create_udwf_expr, get_or_init_udwf};
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// #
+/// # use datafusion_common::arrow::datatypes::Field;
+/// # use datafusion_common::ScalarValue;
+/// # use datafusion_expr::{col, lit};
+/// #
+/// # get_or_init_udwf!(Lead, lead, "user-defined window function");
+/// #
+/// /// Creates `lead(expr, offset, default)` with 3 parameters:
+/// ///
+/// /// ```
+/// /// /// Returns a value evaluated at the row that is offset rows
+/// /// /// after the current row within the partition.
+/// /// pub fn lead(
+/// /// expr: datafusion_expr::Expr,
+/// /// offset: datafusion_expr::Expr,
+/// /// default: datafusion_expr::Expr,
+/// /// ) -> datafusion_expr::Expr {
+/// /// lead_udwf().call(vec![expr, offset, default])
+/// /// }
+/// /// ```
+/// create_udwf_expr!(
+/// Lead,
+/// lead,
+/// [expr, offset, default],
+/// "Returns a value evaluated at the row that is offset rows after the current row within the partition."
+/// );
+/// #
+/// # assert_eq!(
+/// # lead(col("a"), lit(1i64), lit(ScalarValue::Null))
+/// # .name_for_alias()
+/// # .unwrap(),
+/// # "lead(a,Int64(1),NULL) ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
+/// # );
+/// #
+/// # #[derive(Debug)]
+/// # struct Lead {
+/// # signature: Signature,
+/// # }
+/// #
+/// # impl Default for Lead {
+/// # fn default() -> Self {
+/// # Self {
+/// # signature: Signature::one_of(
+/// # vec![
+/// # TypeSignature::Any(1),
+/// # TypeSignature::Any(2),
+/// # TypeSignature::Any(3),
+/// # ],
+/// # Volatility::Immutable,
+/// # ),
+/// # }
+/// # }
+/// # }
+/// #
+/// # impl WindowUDFImpl for Lead {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "lead"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(
+/// # field_args.name(),
+/// # field_args.get_input_type(0).unwrap(),
+/// # false,
+/// # ))
+/// # }
+/// # }
+/// ```
+#[macro_export]
+macro_rules! create_udwf_expr {
+ // zero arguments
+ ($UDWF:ident, $OUT_FN_NAME:ident, $DOC:expr) => {
+ paste::paste! {
+ #[doc = " Create a [`WindowFunction`](datafusion_expr::Expr::WindowFunction) expression for"]
+ #[doc = concat!(" [`", stringify!($UDWF), "`] user-defined window function.")]
+ #[doc = ""]
+ #[doc = concat!(" ", $DOC)]
+ pub fn $OUT_FN_NAME() -> datafusion_expr::Expr {
+ [<$OUT_FN_NAME _udwf>]().call(vec![])
+ }
+ }
+ };
+
+ // 1 or more arguments
+ ($UDWF:ident, $OUT_FN_NAME:ident, [$($PARAM:ident),+], $DOC:expr) => {
+ paste::paste! {
+ #[doc = " Create a [`WindowFunction`](datafusion_expr::Expr::WindowFunction) expression for"]
+ #[doc = concat!(" [`", stringify!($UDWF), "`] user-defined window function.")]
+ #[doc = ""]
+ #[doc = concat!(" ", $DOC)]
+ pub fn $OUT_FN_NAME(
+ $($PARAM: datafusion_expr::Expr),+
+ ) -> datafusion_expr::Expr {
+ [<$OUT_FN_NAME _udwf>]()
+ .call(vec![$($PARAM),+])
+ }
+ }
+ };
+}
+
+/// Defines a user-defined window function.
+///
+/// Combines [`get_or_init_udwf!`] and [`create_udwf_expr!`] into a
+/// single macro for convenience.
+///
+/// # Arguments
+///
+/// * `$UDWF`: The struct which defines the [`Signature`] of the
+/// user-defined window function.
+/// * `$OUT_FN_NAME`: The basename to generate a unique function name like
+/// `$OUT_FN_NAME_udwf`.
+/// * (optional) `[$($PARAM:ident),+]`: An array of 1 or more parameters
+/// for the generated function. The type of parameters is [`Expr`].
+/// When omitted this creates a function with zero parameters.
+/// * `$DOC`: Doc comments for UDWF.
+/// * (optional) `$CTOR`: Pass a custom constructor. When omitted it
+/// automatically resolves to `$UDWF::default()`.
+///
+/// [`Signature`]: datafusion_expr::Signature
+/// [`Expr`]: datafusion_expr::Expr
+///
+/// # Usage
+///
+/// ## Expression API With Zero parameters
+/// 1. Uses default constructor for UDWF.
+///
+/// ```
+/// # use std::any::Any;
+/// # use datafusion_common::arrow::datatypes::{DataType, Field};
+/// # use datafusion_expr::{PartitionEvaluator, Signature, Volatility, WindowUDFImpl};
+/// #
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// # use datafusion_functions_window::{define_udwf_and_expr, get_or_init_udwf, create_udwf_expr};
+/// #
+/// /// 1. Defines the `simple_udwf()` user-defined window function.
+/// ///
+/// /// 2. Defines the expression API:
+/// /// ```
+/// /// pub fn simple() -> datafusion_expr::Expr {
+/// /// simple_udwf().call(vec![])
+/// /// }
+/// /// ```
+/// define_udwf_and_expr!(
+/// SimpleUDWF,
+/// simple,
+/// "a simple user-defined window function"
+/// );
+/// #
+/// # assert_eq!(simple_udwf().name(), "simple_user_defined_window_function");
+/// #
+/// # #[derive(Debug)]
+/// # struct SimpleUDWF {
+/// # signature: Signature,
+/// # }
+/// #
+/// # impl Default for SimpleUDWF {
+/// # fn default() -> Self {
+/// # Self {
+/// # signature: Signature::any(0, Volatility::Immutable),
+/// # }
+/// # }
+/// # }
+/// #
+/// # impl WindowUDFImpl for SimpleUDWF {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "simple_user_defined_window_function"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(field_args.name(), DataType::Int64, false))
+/// # }
+/// # }
+/// #
+/// ```
+///
+/// 2. Uses a custom constructor for UDWF.
+///
+/// ```
+/// # use std::any::Any;
+/// # use datafusion_common::arrow::datatypes::{DataType, Field};
+/// # use datafusion_expr::{PartitionEvaluator, Signature, Volatility, WindowUDFImpl};
+/// # use datafusion_functions_window::{create_udwf_expr, define_udwf_and_expr, get_or_init_udwf};
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// #
+/// /// 1. Defines the `row_number_udwf()` user-defined window function.
+/// ///
+/// /// 2. Defines the expression API:
+/// /// ```
+/// /// pub fn row_number() -> datafusion_expr::Expr {
+/// /// row_number_udwf().call(vec![])
+/// /// }
+/// /// ```
+/// define_udwf_and_expr!(
+/// RowNumber,
+/// row_number,
+/// "Returns a unique row number for each row in window partition beginning at 1.",
+/// RowNumber::new // <-- custom constructor
+/// );
+/// #
+/// # assert_eq!(
+/// # row_number().name_for_alias().unwrap(),
+/// # "row_number() ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
+/// # );
+/// #
+/// # #[derive(Debug)]
+/// # struct RowNumber {
+/// # signature: Signature,
+/// # }
+/// # impl RowNumber {
+/// # fn new() -> Self {
+/// # Self {
+/// # signature: Signature::any(0, Volatility::Immutable),
+/// # }
+/// # }
+/// # }
+/// # impl WindowUDFImpl for RowNumber {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "row_number"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(field_args.name(), DataType::UInt64, false))
+/// # }
+/// # }
+/// ```
+///
+/// ## Expression API With Multiple Parameters
+/// 3. Uses default constructor for UDWF
+///
+/// ```
+/// # use std::any::Any;
+/// #
+/// # use datafusion_expr::{
+/// # PartitionEvaluator, Signature, TypeSignature, Volatility, WindowUDFImpl,
+/// # };
+/// #
+/// # use datafusion_functions_window::{create_udwf_expr, define_udwf_and_expr, get_or_init_udwf};
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// #
+/// # use datafusion_common::arrow::datatypes::Field;
+/// # use datafusion_common::ScalarValue;
+/// # use datafusion_expr::{col, lit};
+/// #
+/// /// 1. Defines the `lead_udwf()` user-defined window function.
+/// ///
+/// /// 2. Defines the expression API:
+/// /// ```
+/// /// pub fn lead(
+/// /// expr: datafusion_expr::Expr,
+/// /// offset: datafusion_expr::Expr,
+/// /// default: datafusion_expr::Expr,
+/// /// ) -> datafusion_expr::Expr {
+/// /// lead_udwf().call(vec![expr, offset, default])
+/// /// }
+/// /// ```
+/// define_udwf_and_expr!(
+/// Lead,
+/// lead,
+/// [expr, offset, default], // <- 3 parameters
+/// "user-defined window function"
+/// );
+/// #
+/// # assert_eq!(
+/// # lead(col("a"), lit(1i64), lit(ScalarValue::Null))
+/// # .name_for_alias()
+/// # .unwrap(),
+/// # "lead(a,Int64(1),NULL) ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
+/// # );
+/// #
+/// # #[derive(Debug)]
+/// # struct Lead {
+/// # signature: Signature,
+/// # }
+/// #
+/// # impl Default for Lead {
+/// # fn default() -> Self {
+/// # Self {
+/// # signature: Signature::one_of(
+/// # vec![
+/// # TypeSignature::Any(1),
+/// # TypeSignature::Any(2),
+/// # TypeSignature::Any(3),
+/// # ],
+/// # Volatility::Immutable,
+/// # ),
+/// # }
+/// # }
+/// # }
+/// #
+/// # impl WindowUDFImpl for Lead {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "lead"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(
+/// # field_args.name(),
+/// # field_args.get_input_type(0).unwrap(),
+/// # false,
+/// # ))
+/// # }
+/// # }
+/// ```
+/// 4. Uses custom constructor for UDWF
+///
+/// ```
+/// # use std::any::Any;
+/// #
+/// # use datafusion_expr::{
+/// # PartitionEvaluator, Signature, TypeSignature, Volatility, WindowUDFImpl,
+/// # };
+/// #
+/// # use datafusion_functions_window::{create_udwf_expr, define_udwf_and_expr, get_or_init_udwf};
+/// # use datafusion_functions_window_common::field::WindowUDFFieldArgs;
+/// #
+/// # use datafusion_common::arrow::datatypes::Field;
+/// # use datafusion_common::ScalarValue;
+/// # use datafusion_expr::{col, lit};
+/// #
+/// /// 1. Defines the `lead_udwf()` user-defined window function.
+/// ///
+/// /// 2. Defines the expression API:
+/// /// ```
+/// /// pub fn lead(
+/// /// expr: datafusion_expr::Expr,
+/// /// offset: datafusion_expr::Expr,
+/// /// default: datafusion_expr::Expr,
+/// /// ) -> datafusion_expr::Expr {
+/// /// lead_udwf().call(vec![expr, offset, default])
+/// /// }
+/// /// ```
+/// define_udwf_and_expr!(
+/// Lead,
+/// lead,
+/// [expr, offset, default], // <- 3 parameters
+/// "user-defined window function",
+/// Lead::new // <- Custom constructor
+/// );
+/// #
+/// # assert_eq!(
+/// # lead(col("a"), lit(1i64), lit(ScalarValue::Null))
+/// # .name_for_alias()
+/// # .unwrap(),
+/// # "lead(a,Int64(1),NULL) ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
+/// # );
+/// #
+/// # #[derive(Debug)]
+/// # struct Lead {
+/// # signature: Signature,
+/// # }
+/// #
+/// # impl Lead {
+/// # fn new() -> Self {
+/// # Self {
+/// # signature: Signature::one_of(
+/// # vec![
+/// # TypeSignature::Any(1),
+/// # TypeSignature::Any(2),
+/// # TypeSignature::Any(3),
+/// # ],
+/// # Volatility::Immutable,
+/// # ),
+/// # }
+/// # }
+/// # }
+/// #
+/// # impl WindowUDFImpl for Lead {
+/// # fn as_any(&self) -> &dyn Any {
+/// # self
+/// # }
+/// # fn name(&self) -> &str {
+/// # "lead"
+/// # }
+/// # fn signature(&self) -> &Signature {
+/// # &self.signature
+/// # }
+/// # fn partition_evaluator(
+/// # &self,
+/// # ) -> datafusion_common::Result<Box<dyn PartitionEvaluator>> {
+/// # unimplemented!()
+/// # }
+/// # fn field(&self, field_args: WindowUDFFieldArgs) -> datafusion_common::Result<Field> {
+/// # Ok(Field::new(
+/// # field_args.name(),
+/// # field_args.get_input_type(0).unwrap(),
+/// # false,
+/// # ))
+/// # }
+/// # }
+/// ```
+#[macro_export]
+macro_rules! define_udwf_and_expr {
+ // Defines UDWF with default constructor
+ // Defines expression API with zero parameters
+ ($UDWF:ident, $OUT_FN_NAME:ident, $DOC:expr) => {
+ get_or_init_udwf!($UDWF, $OUT_FN_NAME, $DOC);
+ create_udwf_expr!($UDWF, $OUT_FN_NAME, $DOC);
+ };
+
+ // Defines UDWF by passing a custom constructor
+ // Defines expression API with zero parameters
+ ($UDWF:ident, $OUT_FN_NAME:ident, $DOC:expr, $CTOR:path) => {
+ get_or_init_udwf!($UDWF, $OUT_FN_NAME, $DOC, $CTOR);
+ create_udwf_expr!($UDWF, $OUT_FN_NAME, $DOC);
+ };
+
+ // Defines UDWF with default constructor
+ // Defines expression API with multiple parameters
+ ($UDWF:ident, $OUT_FN_NAME:ident, [$($PARAM:ident),+], $DOC:expr) => {
+ get_or_init_udwf!($UDWF, $OUT_FN_NAME, $DOC);
+ create_udwf_expr!($UDWF, $OUT_FN_NAME, [$($PARAM),+], $DOC);
+ };
+
+ // Defines UDWF by passing a custom constructor
+ // Defines expression API with multiple parameters
+ ($UDWF:ident, $OUT_FN_NAME:ident, [$($PARAM:ident),+], $DOC:expr, $CTOR:path) => {
+ get_or_init_udwf!($UDWF, $OUT_FN_NAME, $DOC, $CTOR);
+ create_udwf_expr!($UDWF, $OUT_FN_NAME, [$($PARAM),+], $DOC);
+ };
+}
diff --git a/datafusion/functions-window/src/row_number.rs b/datafusion/functions-window/src/row_number.rs
index 7f348bf9d2..c903f6778a 100644
--- a/datafusion/functions-window/src/row_number.rs
+++ b/datafusion/functions-window/src/row_number.rs
@@ -17,41 +17,28 @@
//! Defines physical expression for `row_number` that can evaluated at runtime during query execution
-use std::any::Any;
-use std::fmt::Debug;
-use std::ops::Range;
-
use datafusion_common::arrow::array::ArrayRef;
use datafusion_common::arrow::array::UInt64Array;
use datafusion_common::arrow::compute::SortOptions;
use datafusion_common::arrow::datatypes::DataType;
use datafusion_common::arrow::datatypes::Field;
use datafusion_common::{Result, ScalarValue};
-use datafusion_expr::expr::WindowFunction;
-use datafusion_expr::{Expr, PartitionEvaluator, Signature, Volatility, WindowUDFImpl};
+use datafusion_expr::window_doc_sections::DOC_SECTION_RANKING;
+use datafusion_expr::{
+ Documentation, PartitionEvaluator, Signature, Volatility, WindowUDFImpl,
+};
use datafusion_functions_window_common::field;
use field::WindowUDFFieldArgs;
+use std::any::Any;
+use std::fmt::Debug;
+use std::ops::Range;
+use std::sync::OnceLock;
-/// Create a [`WindowFunction`](Expr::WindowFunction) expression for
-/// `row_number` user-defined window function.
-pub fn row_number() -> Expr {
- Expr::WindowFunction(WindowFunction::new(row_number_udwf(), vec![]))
-}
-
-/// Singleton instance of `row_number`, ensures the UDWF is only created once.
-#[allow(non_upper_case_globals)]
-static STATIC_RowNumber: std::sync::OnceLock<std::sync::Arc<datafusion_expr::WindowUDF>> =
- std::sync::OnceLock::new();
-
-/// Returns a [`WindowUDF`](datafusion_expr::WindowUDF) for `row_number`
-/// user-defined window function.
-pub fn row_number_udwf() -> std::sync::Arc<datafusion_expr::WindowUDF> {
- STATIC_RowNumber
- .get_or_init(|| {
- std::sync::Arc::new(datafusion_expr::WindowUDF::from(RowNumber::default()))
- })
- .clone()
-}
+define_udwf_and_expr!(
+ RowNumber,
+ row_number,
+ "Returns a unique row number for each row in window partition beginning at 1."
+);
/// row_number expression
#[derive(Debug)]
@@ -74,6 +61,21 @@ impl Default for RowNumber {
}
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_row_number_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_RANKING)
+ .with_description(
+ "Number of the current row within its partition, counting from 1.",
+ )
+ .with_syntax_example("row_number()")
+ .build()
+ .unwrap()
+ })
+}
+
impl WindowUDFImpl for RowNumber {
fn as_any(&self) -> &dyn Any {
self
@@ -101,6 +103,10 @@ impl WindowUDFImpl for RowNumber {
nulls_first: false,
})
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_row_number_doc())
+ }
}
/// State for the `row_number` built-in window function.
diff --git a/datafusion/functions/Cargo.toml b/datafusion/functions/Cargo.toml
index ff1b926a9b..a3d114221d 100644
--- a/datafusion/functions/Cargo.toml
+++ b/datafusion/functions/Cargo.toml
@@ -102,6 +102,11 @@ harness = false
name = "to_timestamp"
required-features = ["datetime_expressions"]
+[[bench]]
+harness = false
+name = "encoding"
+required-features = ["encoding_expressions"]
+
[[bench]]
harness = false
name = "regx"
diff --git a/datafusion/functions/benches/encoding.rs b/datafusion/functions/benches/encoding.rs
new file mode 100644
index 0000000000..d49235aac9
--- /dev/null
+++ b/datafusion/functions/benches/encoding.rs
@@ -0,0 +1,53 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+extern crate criterion;
+
+use arrow::util::bench_util::create_string_array_with_len;
+use criterion::{black_box, criterion_group, criterion_main, Criterion};
+use datafusion_expr::ColumnarValue;
+use datafusion_functions::encoding;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let decode = encoding::decode();
+ for size in [1024, 4096, 8192] {
+ let str_array = Arc::new(create_string_array_with_len::<i32>(size, 0.2, 32));
+ c.bench_function(&format!("base64_decode/{size}"), |b| {
+ let method = ColumnarValue::Scalar("base64".into());
+ let encoded = encoding::encode()
+ .invoke(&[ColumnarValue::Array(str_array.clone()), method.clone()])
+ .unwrap();
+
+ let args = vec![encoded, method];
+ b.iter(|| black_box(decode.invoke(&args).unwrap()))
+ });
+
+ c.bench_function(&format!("hex_decode/{size}"), |b| {
+ let method = ColumnarValue::Scalar("hex".into());
+ let encoded = encoding::encode()
+ .invoke(&[ColumnarValue::Array(str_array.clone()), method.clone()])
+ .unwrap();
+
+ let args = vec![encoded, method];
+ b.iter(|| black_box(decode.invoke(&args).unwrap()))
+ });
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
diff --git a/datafusion/functions/src/core/coalesce.rs b/datafusion/functions/src/core/coalesce.rs
index 4eccb5180e..8155c04da6 100644
--- a/datafusion/functions/src/core/coalesce.rs
+++ b/datafusion/functions/src/core/coalesce.rs
@@ -15,17 +15,18 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
use arrow::array::{new_null_array, BooleanArray};
use arrow::compute::kernels::zip::zip;
use arrow::compute::{and, is_not_null, is_null};
use arrow::datatypes::DataType;
use datafusion_common::{exec_err, ExprSchema, Result};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_CONDITIONAL;
use datafusion_expr::type_coercion::binary::type_union_resolution;
-use datafusion_expr::{ColumnarValue, Expr, ExprSchemable};
+use datafusion_expr::{ColumnarValue, Documentation, Expr, ExprSchemable};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use itertools::Itertools;
+use std::any::Any;
+use std::sync::OnceLock;
#[derive(Debug)]
pub struct CoalesceFunc {
@@ -46,6 +47,23 @@ impl CoalesceFunc {
}
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_coalesce_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_CONDITIONAL)
+ .with_description("Returns the first of its arguments that is not _null_. Returns _null_ if all arguments are _null_. This function is often used to substitute a default value for _null_ values.")
+ .with_syntax_example("coalesce(expression1[, ..., expression_n])")
+ .with_argument(
+ "expression1, expression_n",
+ "Expression to use if previous expressions are _null_. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary."
+ )
+ .build()
+ .unwrap()
+ })
+}
+
impl ScalarUDFImpl for CoalesceFunc {
fn as_any(&self) -> &dyn Any {
self
@@ -140,6 +158,10 @@ impl ScalarUDFImpl for CoalesceFunc {
.unwrap_or(arg_types.first().unwrap().clone());
Ok(vec![new_type; arg_types.len()])
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_coalesce_doc())
+ }
}
#[cfg(test)]
diff --git a/datafusion/functions/src/crypto/sha224.rs b/datafusion/functions/src/crypto/sha224.rs
index 2795c4a250..d603e5bcf2 100644
--- a/datafusion/functions/src/crypto/sha224.rs
+++ b/datafusion/functions/src/crypto/sha224.rs
@@ -19,13 +19,18 @@
use super::basic::{sha224, utf8_or_binary_to_binary_type};
use arrow::datatypes::DataType;
use datafusion_common::Result;
-use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_HASHING;
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarUDFImpl, Signature, Volatility,
+};
use std::any::Any;
+use std::sync::OnceLock;
#[derive(Debug)]
pub struct SHA224Func {
signature: Signature,
}
+
impl Default for SHA224Func {
fn default() -> Self {
Self::new()
@@ -44,6 +49,21 @@ impl SHA224Func {
}
}
}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_sha224_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_HASHING)
+ .with_description("Computes the SHA-224 hash of a binary string.")
+ .with_syntax_example("sha224(expression)")
+ .with_standard_argument("expression", "String")
+ .build()
+ .unwrap()
+ })
+}
+
impl ScalarUDFImpl for SHA224Func {
fn as_any(&self) -> &dyn Any {
self
@@ -60,7 +80,12 @@ impl ScalarUDFImpl for SHA224Func {
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
utf8_or_binary_to_binary_type(&arg_types[0], self.name())
}
+
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
sha224(args)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_sha224_doc())
+ }
}
diff --git a/datafusion/functions/src/datetime/to_date.rs b/datafusion/functions/src/datetime/to_date.rs
index ae2ef58899..fc3f062e12 100644
--- a/datafusion/functions/src/datetime/to_date.rs
+++ b/datafusion/functions/src/datetime/to_date.rs
@@ -15,17 +15,19 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
+use crate::datetime::common::*;
use arrow::datatypes::DataType;
use arrow::datatypes::DataType::Date32;
use arrow::error::ArrowError::ParseError;
use arrow::{array::types::Date32Type, compute::kernels::cast_utils::Parser};
-
-use crate::datetime::common::*;
use datafusion_common::error::DataFusionError;
use datafusion_common::{arrow_err, exec_err, internal_datafusion_err, Result};
-use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_DATETIME;
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarUDFImpl, Signature, Volatility,
+};
+use std::any::Any;
+use std::sync::OnceLock;
#[derive(Debug)]
pub struct ToDateFunc {
@@ -77,6 +79,50 @@ impl ToDateFunc {
}
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_to_date_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_DATETIME)
+ .with_description(r#"Converts a value to a date (`YYYY-MM-DD`).
+Supports strings, integer and double types as input.
+Strings are parsed as YYYY-MM-DD (e.g. '2023-07-20') if no [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)s are provided.
+Integers and doubles are interpreted as days since the unix epoch (`1970-01-01T00:00:00Z`).
+Returns the corresponding date.
+
+Note: `to_date` returns Date32, which represents its values as the number of days since unix epoch(`1970-01-01`) stored as signed 32 bit value. The largest supported date value is `9999-12-31`.
+"#)
+ .with_syntax_example("to_date('2017-05-31', '%Y-%m-%d')")
+ .with_sql_example(r#"```sql
+> select to_date('2023-01-31');
++-----------------------------+
+| to_date(Utf8("2023-01-31")) |
++-----------------------------+
+| 2023-01-31 |
++-----------------------------+
+> select to_date('2023/01/31', '%Y-%m-%d', '%Y/%m/%d');
++---------------------------------------------------------------+
+| to_date(Utf8("2023/01/31"),Utf8("%Y-%m-%d"),Utf8("%Y/%m/%d")) |
++---------------------------------------------------------------+
+| 2023-01-31 |
++---------------------------------------------------------------+
+```
+
+Additional examples can be found [here](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/to_date.rs)
+"#)
+ .with_standard_argument("expression", "String")
+ .with_argument(
+ "format_n",
+ "Optional [Chrono format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) strings to use to parse the expression. Formats will be tried in the order
+ they appear with the first successful one being returned. If none of the formats successfully parse the expression
+ an error will be returned.",
+ )
+ .build()
+ .unwrap()
+ })
+}
+
impl ScalarUDFImpl for ToDateFunc {
fn as_any(&self) -> &dyn Any {
self
@@ -117,6 +163,10 @@ impl ScalarUDFImpl for ToDateFunc {
}
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_to_date_doc())
+ }
}
#[cfg(test)]
diff --git a/datafusion/functions/src/encoding/inner.rs b/datafusion/functions/src/encoding/inner.rs
index 19dfca107a..a791d77697 100644
--- a/datafusion/functions/src/encoding/inner.rs
+++ b/datafusion/functions/src/encoding/inner.rs
@@ -18,9 +18,12 @@
//! Encoding expressions
use arrow::{
- array::{Array, ArrayRef, BinaryArray, OffsetSizeTrait, StringArray},
- datatypes::DataType,
+ array::{
+ Array, ArrayRef, BinaryArray, GenericByteArray, OffsetSizeTrait, StringArray,
+ },
+ datatypes::{ByteArrayType, DataType},
};
+use arrow_buffer::{Buffer, OffsetBufferBuilder};
use base64::{engine::general_purpose, Engine as _};
use datafusion_common::{
cast::{as_generic_binary_array, as_generic_string_array},
@@ -28,10 +31,11 @@ use datafusion_common::{
};
use datafusion_common::{exec_err, ScalarValue};
use datafusion_common::{DataFusionError, Result};
-use datafusion_expr::ColumnarValue;
-use std::sync::Arc;
+use datafusion_expr::{ColumnarValue, Documentation};
+use std::sync::{Arc, OnceLock};
use std::{fmt, str::FromStr};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_BINARY_STRING;
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
@@ -54,6 +58,22 @@ impl EncodeFunc {
}
}
+static ENCODE_DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_encode_doc() -> &'static Documentation {
+ ENCODE_DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_BINARY_STRING)
+ .with_description("Encode binary data into a textual representation.")
+ .with_syntax_example("encode(expression, format)")
+ .with_argument("expression", "Expression containing string or binary data")
+ .with_argument("format", "Supported formats are: `base64`, `hex`")
+ .with_related_udf("decode")
+ .build()
+ .unwrap()
+ })
+}
+
impl ScalarUDFImpl for EncodeFunc {
fn as_any(&self) -> &dyn Any {
self
@@ -100,6 +120,10 @@ impl ScalarUDFImpl for EncodeFunc {
),
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_encode_doc())
+ }
}
#[derive(Debug)]
@@ -121,6 +145,22 @@ impl DecodeFunc {
}
}
+static DECODE_DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_decode_doc() -> &'static Documentation {
+ DECODE_DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_BINARY_STRING)
+ .with_description("Decode binary data from textual representation in string.")
+ .with_syntax_example("decode(expression, format)")
+ .with_argument("expression", "Expression containing encoded string data")
+ .with_argument("format", "Same arguments as [encode](#encode)")
+ .with_related_udf("encode")
+ .build()
+ .unwrap()
+ })
+}
+
impl ScalarUDFImpl for DecodeFunc {
fn as_any(&self) -> &dyn Any {
self
@@ -167,6 +207,10 @@ impl ScalarUDFImpl for DecodeFunc {
),
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_decode_doc())
+ }
}
#[derive(Debug, Copy, Clone)]
@@ -245,16 +289,22 @@ fn base64_encode(input: &[u8]) -> String {
general_purpose::STANDARD_NO_PAD.encode(input)
}
-fn hex_decode(input: &[u8]) -> Result<Vec<u8>> {
- hex::decode(input).map_err(|e| {
+fn hex_decode(input: &[u8], buf: &mut [u8]) -> Result<usize> {
+ // only write input / 2 bytes to buf
+ let out_len = input.len() / 2;
+ let buf = &mut buf[..out_len];
+ hex::decode_to_slice(input, buf).map_err(|e| {
DataFusionError::Internal(format!("Failed to decode from hex: {}", e))
- })
+ })?;
+ Ok(out_len)
}
-fn base64_decode(input: &[u8]) -> Result<Vec<u8>> {
- general_purpose::STANDARD_NO_PAD.decode(input).map_err(|e| {
- DataFusionError::Internal(format!("Failed to decode from base64: {}", e))
- })
+fn base64_decode(input: &[u8], buf: &mut [u8]) -> Result<usize> {
+ general_purpose::STANDARD_NO_PAD
+ .decode_slice(input, buf)
+ .map_err(|e| {
+ DataFusionError::Internal(format!("Failed to decode from base64: {}", e))
+ })
}
macro_rules! encode_to_array {
@@ -267,14 +317,35 @@ macro_rules! encode_to_array {
}};
}
-macro_rules! decode_to_array {
- ($METHOD: ident, $INPUT:expr) => {{
- let binary_array: BinaryArray = $INPUT
- .iter()
- .map(|x| x.map(|x| $METHOD(x.as_ref())).transpose())
- .collect::<Result<_>>()?;
- Arc::new(binary_array)
- }};
+fn decode_to_array<F, T: ByteArrayType>(
+ method: F,
+ input: &GenericByteArray<T>,
+ conservative_upper_bound_size: usize,
+) -> Result<ArrayRef>
+where
+ F: Fn(&[u8], &mut [u8]) -> Result<usize>,
+{
+ let mut values = vec![0; conservative_upper_bound_size];
+ let mut offsets = OffsetBufferBuilder::new(input.len());
+ let mut total_bytes_decoded = 0;
+ for v in input {
+ if let Some(v) = v {
+ let cursor = &mut values[total_bytes_decoded..];
+ let decoded = method(v.as_ref(), cursor)?;
+ total_bytes_decoded += decoded;
+ offsets.push_length(decoded);
+ } else {
+ offsets.push_length(0);
+ }
+ }
+ // We reserved an upper bound size for the values buffer, but we only use the actual size
+ values.truncate(total_bytes_decoded);
+ let binary_array = BinaryArray::try_new(
+ offsets.finish(),
+ Buffer::from_vec(values),
+ input.nulls().cloned(),
+ )?;
+ Ok(Arc::new(binary_array))
}
impl Encoding {
@@ -381,10 +452,7 @@ impl Encoding {
T: OffsetSizeTrait,
{
let input_value = as_generic_binary_array::<T>(value)?;
- let array: ArrayRef = match self {
- Self::Base64 => decode_to_array!(base64_decode, input_value),
- Self::Hex => decode_to_array!(hex_decode, input_value),
- };
+ let array = self.decode_byte_array(input_value)?;
Ok(ColumnarValue::Array(array))
}
@@ -393,12 +461,29 @@ impl Encoding {
T: OffsetSizeTrait,
{
let input_value = as_generic_string_array::<T>(value)?;
- let array: ArrayRef = match self {
- Self::Base64 => decode_to_array!(base64_decode, input_value),
- Self::Hex => decode_to_array!(hex_decode, input_value),
- };
+ let array = self.decode_byte_array(input_value)?;
Ok(ColumnarValue::Array(array))
}
+
+ fn decode_byte_array<T: ByteArrayType>(
+ &self,
+ input_value: &GenericByteArray<T>,
+ ) -> Result<ArrayRef> {
+ match self {
+ Self::Base64 => {
+ let upper_bound =
+ base64::decoded_len_estimate(input_value.values().len());
+ decode_to_array(base64_decode, input_value, upper_bound)
+ }
+ Self::Hex => {
+ // Calculate the upper bound for decoded byte size
+ // For hex encoding, each pair of hex characters (2 bytes) represents 1 byte when decoded
+ // So the upper bound is half the length of the input values.
+ let upper_bound = input_value.values().len() / 2;
+ decode_to_array(hex_decode, input_value, upper_bound)
+ }
+ }
+ }
}
impl fmt::Display for Encoding {
diff --git a/datafusion/functions/src/lib.rs b/datafusion/functions/src/lib.rs
index bb680f3c67..81be555266 100644
--- a/datafusion/functions/src/lib.rs
+++ b/datafusion/functions/src/lib.rs
@@ -92,9 +92,6 @@ pub mod macros;
pub mod string;
make_stub_package!(string, "string_expressions");
-#[cfg(feature = "string_expressions")]
-mod regexp_common;
-
/// Core datafusion expressions
/// Enabled via feature flag `core_expressions`
#[cfg(feature = "core_expressions")]
diff --git a/datafusion/functions/src/macros.rs b/datafusion/functions/src/macros.rs
index e47818bc86..e850673ef8 100644
--- a/datafusion/functions/src/macros.rs
+++ b/datafusion/functions/src/macros.rs
@@ -284,7 +284,7 @@ macro_rules! make_math_binary_udf {
use arrow::datatypes::DataType;
use datafusion_common::{exec_err, DataFusionError, Result};
use datafusion_expr::sort_properties::{ExprProperties, SortProperties};
- use datafusion_expr::TypeSignature::*;
+ use datafusion_expr::TypeSignature;
use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
#[derive(Debug)]
@@ -298,8 +298,8 @@ macro_rules! make_math_binary_udf {
Self {
signature: Signature::one_of(
vec![
- Exact(vec![Float32, Float32]),
- Exact(vec![Float64, Float64]),
+ TypeSignature::Exact(vec![Float32, Float32]),
+ TypeSignature::Exact(vec![Float64, Float64]),
],
Volatility::Immutable,
),
diff --git a/datafusion/functions/src/math/log.rs b/datafusion/functions/src/math/log.rs
index b6f01f5e46..1ffda8759d 100644
--- a/datafusion/functions/src/math/log.rs
+++ b/datafusion/functions/src/math/log.rs
@@ -18,7 +18,7 @@
//! Math function: `log()`.
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use super::power::PowerFunc;
@@ -29,9 +29,12 @@ use datafusion_common::{
ScalarValue,
};
use datafusion_expr::expr::ScalarFunction;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_MATH;
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyInfo};
use datafusion_expr::sort_properties::{ExprProperties, SortProperties};
-use datafusion_expr::{lit, ColumnarValue, Expr, ScalarUDF, TypeSignature::*};
+use datafusion_expr::{
+ lit, ColumnarValue, Documentation, Expr, ScalarUDF, TypeSignature::*,
+};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
#[derive(Debug)]
@@ -45,6 +48,22 @@ impl Default for LogFunc {
}
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_log_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_MATH)
+ .with_description("Returns the base-x logarithm of a number. Can either provide a specified base, or if omitted then takes the base-10 of a number.")
+ .with_syntax_example(r#"log(base, numeric_expression)
+log(numeric_expression)"#)
+ .with_standard_argument("base", "Base numeric")
+ .with_standard_argument("numeric_expression", "Numeric")
+ .build()
+ .unwrap()
+ })
+}
+
impl LogFunc {
pub fn new() -> Self {
use DataType::*;
@@ -168,6 +187,10 @@ impl ScalarUDFImpl for LogFunc {
Ok(ColumnarValue::Array(arr))
}
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_log_doc())
+ }
+
/// Simplify the `log` function by the relevant rules:
/// 1. Log(a, 1) ===> 0
/// 2. Log(a, Power(a, b)) ===> b
diff --git a/datafusion/functions/src/math/nans.rs b/datafusion/functions/src/math/nans.rs
index 2bd704a7de..b02839b40b 100644
--- a/datafusion/functions/src/math/nans.rs
+++ b/datafusion/functions/src/math/nans.rs
@@ -19,10 +19,9 @@
use arrow::datatypes::DataType;
use datafusion_common::{exec_err, DataFusionError, Result};
-use datafusion_expr::ColumnarValue;
+use datafusion_expr::{ColumnarValue, TypeSignature};
use arrow::array::{ArrayRef, BooleanArray, Float32Array, Float64Array};
-use datafusion_expr::TypeSignature::*;
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
use std::sync::Arc;
@@ -43,7 +42,10 @@ impl IsNanFunc {
use DataType::*;
Self {
signature: Signature::one_of(
- vec![Exact(vec![Float32]), Exact(vec![Float64])],
+ vec![
+ TypeSignature::Exact(vec![Float32]),
+ TypeSignature::Exact(vec![Float64]),
+ ],
Volatility::Immutable,
),
}
diff --git a/datafusion/functions/src/math/power.rs b/datafusion/functions/src/math/power.rs
index 5b790fb56d..831f983d59 100644
--- a/datafusion/functions/src/math/power.rs
+++ b/datafusion/functions/src/math/power.rs
@@ -25,10 +25,9 @@ use datafusion_common::{
};
use datafusion_expr::expr::ScalarFunction;
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyInfo};
-use datafusion_expr::{ColumnarValue, Expr, ScalarUDF};
+use datafusion_expr::{ColumnarValue, Expr, ScalarUDF, TypeSignature};
use arrow::array::{ArrayRef, Float64Array, Int64Array};
-use datafusion_expr::TypeSignature::*;
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
use std::sync::Arc;
@@ -52,7 +51,10 @@ impl PowerFunc {
use DataType::*;
Self {
signature: Signature::one_of(
- vec![Exact(vec![Int64, Int64]), Exact(vec![Float64, Float64])],
+ vec![
+ TypeSignature::Exact(vec![Int64, Int64]),
+ TypeSignature::Exact(vec![Float64, Float64]),
+ ],
Volatility::Immutable,
),
aliases: vec![String::from("pow")],
diff --git a/datafusion/functions/src/regex/regexplike.rs b/datafusion/functions/src/regex/regexplike.rs
index 4cff633a17..61ef35bb6e 100644
--- a/datafusion/functions/src/regex/regexplike.rs
+++ b/datafusion/functions/src/regex/regexplike.rs
@@ -16,7 +16,7 @@
// under the License.
//! Regx expressions
-use arrow::array::{Array, ArrayRef, OffsetSizeTrait};
+use arrow::array::{Array, ArrayRef, GenericStringArray, OffsetSizeTrait};
use arrow::compute::kernels::regexp;
use arrow::datatypes::DataType;
use datafusion_common::exec_err;
@@ -25,32 +25,71 @@ use datafusion_common::{arrow_datafusion_err, plan_err};
use datafusion_common::{
cast::as_generic_string_array, internal_err, DataFusionError, Result,
};
-use datafusion_expr::ColumnarValue;
-use datafusion_expr::TypeSignature::*;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_REGEX;
+use datafusion_expr::{ColumnarValue, Documentation, TypeSignature};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
#[derive(Debug)]
pub struct RegexpLikeFunc {
signature: Signature,
}
+
impl Default for RegexpLikeFunc {
fn default() -> Self {
Self::new()
}
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_regexp_like_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_REGEX)
+ .with_description("Returns true if a [regular expression](https://docs.rs/regex/latest/regex/#syntax) has at least one match in a string, false otherwise.")
+ .with_syntax_example("regexp_like(str, regexp[, flags])")
+ .with_sql_example(r#"```sql
+select regexp_like('Köln', '[a-zA-Z]ö[a-zA-Z]{2}');
++--------------------------------------------------------+
+| regexp_like(Utf8("Köln"),Utf8("[a-zA-Z]ö[a-zA-Z]{2}")) |
++--------------------------------------------------------+
+| true |
++--------------------------------------------------------+
+SELECT regexp_like('aBc', '(b|d)', 'i');
++--------------------------------------------------+
+| regexp_like(Utf8("aBc"),Utf8("(b|d)"),Utf8("i")) |
++--------------------------------------------------+
+| true |
++--------------------------------------------------+
+```
+Additional examples can be found [here](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/regexp.rs)
+"#)
+ .with_standard_argument("str", "String")
+ .with_standard_argument("regexp","Regular")
+ .with_argument("flags",
+ r#"Optional regular expression flags that control the behavior of the regular expression. The following flags are supported:
+ - **i**: case-insensitive: letters match both upper and lower case
+ - **m**: multi-line mode: ^ and $ match begin/end of line
+ - **s**: allow . to match \n
+ - **R**: enables CRLF mode: when multi-line mode is enabled, \r\n is used
+ - **U**: swap the meaning of x* and x*?"#)
+ .build()
+ .unwrap()
+ })
+}
+
impl RegexpLikeFunc {
pub fn new() -> Self {
use DataType::*;
Self {
signature: Signature::one_of(
vec![
- Exact(vec![Utf8, Utf8]),
- Exact(vec![LargeUtf8, Utf8]),
- Exact(vec![Utf8, Utf8, Utf8]),
- Exact(vec![LargeUtf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8]),
+ TypeSignature::Exact(vec![LargeUtf8, LargeUtf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![LargeUtf8, LargeUtf8, LargeUtf8]),
],
Volatility::Immutable,
),
@@ -105,6 +144,10 @@ impl ScalarUDFImpl for RegexpLikeFunc {
result.map(ColumnarValue::Array)
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_regexp_like_doc())
+ }
}
fn regexp_like_func(args: &[ArrayRef]) -> Result<ArrayRef> {
match args[0].data_type() {
@@ -162,7 +205,8 @@ pub fn regexp_like<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
2 => {
let values = as_generic_string_array::<T>(&args[0])?;
let regex = as_generic_string_array::<T>(&args[1])?;
- let array = regexp::regexp_is_match_utf8(values, regex, None)
+ let flags: Option<&GenericStringArray<T>> = None;
+ let array = regexp::regexp_is_match(values, regex, flags)
.map_err(|e| arrow_datafusion_err!(e))?;
Ok(Arc::new(array) as ArrayRef)
@@ -176,7 +220,7 @@ pub fn regexp_like<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
return plan_err!("regexp_like() does not support the \"global\" option");
}
- let array = regexp::regexp_is_match_utf8(values, regex, Some(flags))
+ let array = regexp::regexp_is_match(values, regex, Some(flags))
.map_err(|e| arrow_datafusion_err!(e))?;
Ok(Arc::new(array) as ArrayRef)
diff --git a/datafusion/functions/src/regex/regexpmatch.rs b/datafusion/functions/src/regex/regexpmatch.rs
index a2c314e376..714e2c6a53 100644
--- a/datafusion/functions/src/regex/regexpmatch.rs
+++ b/datafusion/functions/src/regex/regexpmatch.rs
@@ -26,8 +26,7 @@ use datafusion_common::{arrow_datafusion_err, plan_err};
use datafusion_common::{
cast::as_generic_string_array, internal_err, DataFusionError, Result,
};
-use datafusion_expr::ColumnarValue;
-use datafusion_expr::TypeSignature::*;
+use datafusion_expr::{ColumnarValue, TypeSignature};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
use std::sync::Arc;
@@ -53,10 +52,10 @@ impl RegexpMatchFunc {
// For example, given input `(Utf8View, Utf8)`, it first tries coercing to `(Utf8, Utf8)`.
// If that fails, it proceeds to `(LargeUtf8, Utf8)`.
// TODO: Native support Utf8View for regexp_match.
- Exact(vec![Utf8, Utf8]),
- Exact(vec![LargeUtf8, Utf8]),
- Exact(vec![Utf8, Utf8, Utf8]),
- Exact(vec![LargeUtf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8]),
+ TypeSignature::Exact(vec![LargeUtf8, LargeUtf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![LargeUtf8, LargeUtf8, LargeUtf8]),
],
Volatility::Immutable,
),
@@ -131,7 +130,7 @@ pub fn regexp_match<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
let flags = as_generic_string_array::<T>(&args[2])?;
if flags.iter().any(|s| s == Some("g")) {
- return plan_err!("regexp_match() does not support the \"global\" option")
+ return plan_err!("regexp_match() does not support the \"global\" option");
}
regexp::regexp_match(values, regex, Some(flags))
diff --git a/datafusion/functions/src/regex/regexpreplace.rs b/datafusion/functions/src/regex/regexpreplace.rs
index 5c885de202..27800af347 100644
--- a/datafusion/functions/src/regex/regexpreplace.rs
+++ b/datafusion/functions/src/regex/regexpreplace.rs
@@ -33,7 +33,7 @@ use datafusion_common::{
};
use datafusion_expr::function::Hint;
use datafusion_expr::ColumnarValue;
-use datafusion_expr::TypeSignature::*;
+use datafusion_expr::TypeSignature;
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use regex::Regex;
use std::any::Any;
@@ -56,10 +56,10 @@ impl RegexpReplaceFunc {
Self {
signature: Signature::one_of(
vec![
- Exact(vec![Utf8, Utf8, Utf8]),
- Exact(vec![Utf8View, Utf8, Utf8]),
- Exact(vec![Utf8, Utf8, Utf8, Utf8]),
- Exact(vec![Utf8View, Utf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8View, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8, Utf8, Utf8, Utf8]),
+ TypeSignature::Exact(vec![Utf8View, Utf8, Utf8, Utf8]),
],
Volatility::Immutable,
),
diff --git a/datafusion/functions/src/regexp_common.rs b/datafusion/functions/src/regexp_common.rs
deleted file mode 100644
index 748c1a294f..0000000000
--- a/datafusion/functions/src/regexp_common.rs
+++ /dev/null
@@ -1,123 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Common utilities for implementing regex functions
-
-use crate::string::common::StringArrayType;
-
-use arrow::array::{Array, ArrayDataBuilder, BooleanArray};
-use arrow::datatypes::DataType;
-use arrow_buffer::{BooleanBufferBuilder, NullBuffer};
-use datafusion_common::DataFusionError;
-use regex::Regex;
-
-use std::collections::HashMap;
-
-#[cfg(doc)]
-use arrow::array::{LargeStringArray, StringArray, StringViewArray};
-/// Perform SQL `array ~ regex_array` operation on
-/// [`StringArray`] / [`LargeStringArray`] / [`StringViewArray`].
-///
-/// If `regex_array` element has an empty value, the corresponding result value is always true.
-///
-/// `flags_array` are optional [`StringArray`] / [`LargeStringArray`] / [`StringViewArray`] flag,
-/// which allow special search modes, such as case-insensitive and multi-line mode.
-/// See the documentation [here](https://docs.rs/regex/1.5.4/regex/#grouping-and-flags)
-/// for more information.
-///
-/// It is inspired / copied from `regexp_is_match_utf8` [arrow-rs].
-///
-/// Can remove when <https://github.com/apache/arrow-rs/issues/6370> is implemented upstream
-///
-/// [arrow-rs]: https://github.com/apache/arrow-rs/blob/8c956a9f9ab26c14072740cce64c2b99cb039b13/arrow-string/src/regexp.rs#L31-L37
-pub fn regexp_is_match_utf8<'a, S1, S2, S3>(
- array: &'a S1,
- regex_array: &'a S2,
- flags_array: Option<&'a S3>,
-) -> datafusion_common::Result<BooleanArray, DataFusionError>
-where
- &'a S1: StringArrayType<'a>,
- &'a S2: StringArrayType<'a>,
- &'a S3: StringArrayType<'a>,
-{
- if array.len() != regex_array.len() {
- return Err(DataFusionError::Execution(
- "Cannot perform comparison operation on arrays of different length"
- .to_string(),
- ));
- }
-
- let nulls = NullBuffer::union(array.nulls(), regex_array.nulls());
-
- let mut patterns: HashMap<String, Regex> = HashMap::new();
- let mut result = BooleanBufferBuilder::new(array.len());
-
- let complete_pattern = match flags_array {
- Some(flags) => Box::new(regex_array.iter().zip(flags.iter()).map(
- |(pattern, flags)| {
- pattern.map(|pattern| match flags {
- Some(flag) => format!("(?{flag}){pattern}"),
- None => pattern.to_string(),
- })
- },
- )) as Box<dyn Iterator<Item = Option<String>>>,
- None => Box::new(
- regex_array
- .iter()
- .map(|pattern| pattern.map(|pattern| pattern.to_string())),
- ),
- };
-
- array
- .iter()
- .zip(complete_pattern)
- .map(|(value, pattern)| {
- match (value, pattern) {
- (Some(_), Some(pattern)) if pattern == *"" => {
- result.append(true);
- }
- (Some(value), Some(pattern)) => {
- let existing_pattern = patterns.get(&pattern);
- let re = match existing_pattern {
- Some(re) => re,
- None => {
- let re = Regex::new(pattern.as_str()).map_err(|e| {
- DataFusionError::Execution(format!(
- "Regular expression did not compile: {e:?}"
- ))
- })?;
- patterns.entry(pattern).or_insert(re)
- }
- };
- result.append(re.is_match(value));
- }
- _ => result.append(false),
- }
- Ok(())
- })
- .collect::<datafusion_common::Result<Vec<()>, DataFusionError>>()?;
-
- let data = unsafe {
- ArrayDataBuilder::new(DataType::Boolean)
- .len(array.len())
- .buffers(vec![result.into()])
- .nulls(nulls)
- .build_unchecked()
- };
-
- Ok(BooleanArray::from(data))
-}
diff --git a/datafusion/functions/src/string/ascii.rs b/datafusion/functions/src/string/ascii.rs
index 526803eb36..610366b6e6 100644
--- a/datafusion/functions/src/string/ascii.rs
+++ b/datafusion/functions/src/string/ascii.rs
@@ -20,10 +20,11 @@ use arrow::array::{ArrayAccessor, ArrayIter, ArrayRef, AsArray, Int32Array};
use arrow::datatypes::DataType;
use arrow::error::ArrowError;
use datafusion_common::{internal_err, Result};
-use datafusion_expr::ColumnarValue;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
#[derive(Debug)]
pub struct AsciiFunc {
@@ -38,13 +39,8 @@ impl Default for AsciiFunc {
impl AsciiFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::uniform(
- 1,
- vec![Utf8, LargeUtf8, Utf8View],
- Volatility::Immutable,
- ),
+ signature: Signature::string(1, Volatility::Immutable),
}
}
}
@@ -71,6 +67,43 @@ impl ScalarUDFImpl for AsciiFunc {
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
make_scalar_function(ascii, vec![])(args)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_ascii_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_ascii_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description(
+ "Returns the Unicode character code of the first character in a string.",
+ )
+ .with_syntax_example("ascii(str)")
+ .with_sql_example(
+ r#"```sql
+> select ascii('abc');
++--------------------+
+| ascii(Utf8("abc")) |
++--------------------+
+| 97 |
++--------------------+
+> select ascii('🚀');
++-------------------+
+| ascii(Utf8("🚀")) |
++-------------------+
+| 128640 |
++-------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_related_udf("chr")
+ .build()
+ .unwrap()
+ })
}
fn calculate_ascii<'a, V>(array: V) -> Result<ArrayRef, ArrowError>
diff --git a/datafusion/functions/src/string/bit_length.rs b/datafusion/functions/src/string/bit_length.rs
index c37851443c..93d4fa25ae 100644
--- a/datafusion/functions/src/string/bit_length.rs
+++ b/datafusion/functions/src/string/bit_length.rs
@@ -15,17 +15,17 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
use arrow::compute::kernels::length::bit_length;
use arrow::datatypes::DataType;
+use std::any::Any;
+use std::sync::OnceLock;
+use crate::utils::utf8_to_int_type;
use datafusion_common::{exec_err, Result, ScalarValue};
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::utils::utf8_to_int_type;
-
#[derive(Debug)]
pub struct BitLengthFunc {
signature: Signature,
@@ -39,13 +39,8 @@ impl Default for BitLengthFunc {
impl BitLengthFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::uniform(
- 1,
- vec![Utf8, LargeUtf8],
- Volatility::Immutable,
- ),
+ signature: Signature::string(1, Volatility::Immutable),
}
}
}
@@ -88,4 +83,34 @@ impl ScalarUDFImpl for BitLengthFunc {
},
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_bit_length_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_bit_length_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Returns the bit length of a string.")
+ .with_syntax_example("bit_length(str)")
+ .with_sql_example(
+ r#"```sql
+> select bit_length('datafusion');
++--------------------------------+
+| bit_length(Utf8("datafusion")) |
++--------------------------------+
+| 80 |
++--------------------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_related_udf("length")
+ .with_related_udf("octet_length")
+ .build()
+ .unwrap()
+ })
}
diff --git a/datafusion/functions/src/string/btrim.rs b/datafusion/functions/src/string/btrim.rs
index 0e992ff27f..82b7599f07 100644
--- a/datafusion/functions/src/string/btrim.rs
+++ b/datafusion/functions/src/string/btrim.rs
@@ -15,18 +15,18 @@
// specific language governing permissions and limitations
// under the License.
+use crate::string::common::*;
+use crate::utils::{make_scalar_function, utf8_to_str_type};
use arrow::array::{ArrayRef, OffsetSizeTrait};
use arrow::datatypes::DataType;
-use std::any::Any;
-
use datafusion_common::{exec_err, Result};
use datafusion_expr::function::Hint;
-use datafusion_expr::TypeSignature::*;
-use datafusion_expr::{ColumnarValue, Volatility};
-use datafusion_expr::{ScalarUDFImpl, Signature};
-
-use crate::string::common::*;
-use crate::utils::{make_scalar_function, utf8_to_str_type};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarUDFImpl, Signature, TypeSignature, Volatility,
+};
+use std::any::Any;
+use std::sync::OnceLock;
/// Returns the longest string with leading and trailing characters removed. If the characters are not specified, whitespace is removed.
/// btrim('xyxtrimyyx', 'xyz') = 'trim'
@@ -49,18 +49,9 @@ impl Default for BTrimFunc {
impl BTrimFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
signature: Signature::one_of(
- vec![
- // Planner attempts coercion to the target type starting with the most preferred candidate.
- // For example, given input `(Utf8View, Utf8)`, it first tries coercing to `(Utf8View, Utf8View)`.
- // If that fails, it proceeds to `(Utf8, Utf8)`.
- Exact(vec![Utf8View, Utf8View]),
- Exact(vec![Utf8, Utf8]),
- Exact(vec![Utf8View]),
- Exact(vec![Utf8]),
- ],
+ vec![TypeSignature::String(2), TypeSignature::String(1)],
Volatility::Immutable,
),
aliases: vec![String::from("trim")],
@@ -109,6 +100,35 @@ impl ScalarUDFImpl for BTrimFunc {
fn aliases(&self) -> &[String] {
&self.aliases
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_btrim_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_btrim_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Trims the specified trim string from the start and end of a string. If no trim string is provided, all whitespace is removed from the start and end of the input string.")
+ .with_syntax_example("btrim(str[, trim_str])")
+ .with_sql_example(r#"```sql
+> select btrim('__datafusion____', '_');
++-------------------------------------------+
+| btrim(Utf8("__datafusion____"),Utf8("_")) |
++-------------------------------------------+
+| datafusion |
++-------------------------------------------+
+```"#)
+ .with_standard_argument("str", "String")
+ .with_argument("trim_str", "String expression to operate on. Can be a constant, column, or function, and any combination of operators. _Default is whitespace characters._")
+ .with_related_udf("ltrim")
+ .with_related_udf("rtrim")
+ .build()
+ .unwrap()
+ })
}
#[cfg(test)]
diff --git a/datafusion/functions/src/string/chr.rs b/datafusion/functions/src/string/chr.rs
index 4da7dc0159..ae0900af37 100644
--- a/datafusion/functions/src/string/chr.rs
+++ b/datafusion/functions/src/string/chr.rs
@@ -16,7 +16,7 @@
// under the License.
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use arrow::array::ArrayRef;
use arrow::array::StringArray;
@@ -24,13 +24,13 @@ use arrow::datatypes::DataType;
use arrow::datatypes::DataType::Int64;
use arrow::datatypes::DataType::Utf8;
+use crate::utils::make_scalar_function;
use datafusion_common::cast::as_int64_array;
use datafusion_common::{exec_err, Result};
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::utils::make_scalar_function;
-
/// Returns the character with the given code. chr(0) is disallowed because text data types cannot store that character.
/// chr(65) = 'A'
pub fn chr(args: &[ArrayRef]) -> Result<ArrayRef> {
@@ -99,4 +99,35 @@ impl ScalarUDFImpl for ChrFunc {
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
make_scalar_function(chr, vec![])(args)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_chr_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_chr_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description(
+ "Returns the character with the specified ASCII or Unicode code value.",
+ )
+ .with_syntax_example("chr(expression)")
+ .with_sql_example(
+ r#"```sql
+> select chr(128640);
++--------------------+
+| chr(Int64(128640)) |
++--------------------+
+| 🚀 |
++--------------------+
+```"#,
+ )
+ .with_standard_argument("expression", "String")
+ .with_related_udf("ascii")
+ .build()
+ .unwrap()
+ })
}
diff --git a/datafusion/functions/src/string/concat.rs b/datafusion/functions/src/string/concat.rs
index 8a9fae7e9f..e854ff3755 100644
--- a/datafusion/functions/src/string/concat.rs
+++ b/datafusion/functions/src/string/concat.rs
@@ -18,18 +18,18 @@
use arrow::array::{as_largestring_array, Array};
use arrow::datatypes::DataType;
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
+use crate::string::common::*;
+use crate::string::concat;
use datafusion_common::cast::{as_string_array, as_string_view_array};
use datafusion_common::{internal_err, plan_err, Result, ScalarValue};
use datafusion_expr::expr::ScalarFunction;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyInfo};
-use datafusion_expr::{lit, ColumnarValue, Expr, Volatility};
+use datafusion_expr::{lit, ColumnarValue, Documentation, Expr, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::string::common::*;
-use crate::string::concat;
-
#[derive(Debug)]
pub struct ConcatFunc {
signature: Signature,
@@ -248,6 +248,36 @@ impl ScalarUDFImpl for ConcatFunc {
) -> Result<ExprSimplifyResult> {
simplify_concat(args)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_concat_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_concat_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Concatenates multiple strings together.")
+ .with_syntax_example("concat(str[, ..., str_n])")
+ .with_sql_example(
+ r#"```sql
+> select concat('data', 'f', 'us', 'ion');
++-------------------------------------------------------+
+| concat(Utf8("data"),Utf8("f"),Utf8("us"),Utf8("ion")) |
++-------------------------------------------------------+
+| datafusion |
++-------------------------------------------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_argument("str_n", "Subsequent string expressions to concatenate.")
+ .with_related_udf("concat_ws")
+ .build()
+ .unwrap()
+ })
}
pub fn simplify_concat(args: Vec<Expr>) -> Result<ExprSimplifyResult> {
diff --git a/datafusion/functions/src/string/concat_ws.rs b/datafusion/functions/src/string/concat_ws.rs
index 6503a54e65..761dddd104 100644
--- a/datafusion/functions/src/string/concat_ws.rs
+++ b/datafusion/functions/src/string/concat_ws.rs
@@ -17,7 +17,7 @@
use arrow::array::{as_largestring_array, Array, StringArray};
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use arrow::datatypes::DataType;
@@ -27,8 +27,9 @@ use crate::string::concat_ws;
use datafusion_common::cast::{as_string_array, as_string_view_array};
use datafusion_common::{exec_err, internal_err, plan_err, Result, ScalarValue};
use datafusion_expr::expr::ScalarFunction;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyInfo};
-use datafusion_expr::{lit, ColumnarValue, Expr, Volatility};
+use datafusion_expr::{lit, ColumnarValue, Documentation, Expr, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
#[derive(Debug)]
@@ -279,6 +280,45 @@ impl ScalarUDFImpl for ConcatWsFunc {
_ => Ok(ExprSimplifyResult::Original(args)),
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_concat_ws_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_concat_ws_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description(
+ "Concatenates multiple strings together with a specified separator.",
+ )
+ .with_syntax_example("concat_ws(separator, str[, ..., str_n])")
+ .with_sql_example(
+ r#"```sql
+> select concat_ws('_', 'data', 'fusion');
++--------------------------------------------------+
+| concat_ws(Utf8("_"),Utf8("data"),Utf8("fusion")) |
++--------------------------------------------------+
+| data_fusion |
++--------------------------------------------------+
+```"#,
+ )
+ .with_argument(
+ "separator",
+ "Separator to insert between concatenated strings.",
+ )
+ .with_standard_argument("str", "String")
+ .with_standard_argument(
+ "str_n",
+ "Subsequent string expressions to concatenate.",
+ )
+ .with_related_udf("concat")
+ .build()
+ .unwrap()
+ })
}
fn simplify_concat_ws(delimiter: &Expr, args: &[Expr]) -> Result<ExprSimplifyResult> {
diff --git a/datafusion/functions/src/string/contains.rs b/datafusion/functions/src/string/contains.rs
index 4133552160..0f75731aa1 100644
--- a/datafusion/functions/src/string/contains.rs
+++ b/datafusion/functions/src/string/contains.rs
@@ -15,21 +15,20 @@
// specific language governing permissions and limitations
// under the License.
-use crate::regexp_common::regexp_is_match_utf8;
use crate::utils::make_scalar_function;
-
use arrow::array::{Array, ArrayRef, AsArray, GenericStringArray, StringViewArray};
+use arrow::compute::regexp_is_match;
use arrow::datatypes::DataType;
use arrow::datatypes::DataType::{Boolean, LargeUtf8, Utf8, Utf8View};
use datafusion_common::exec_err;
use datafusion_common::DataFusionError;
use datafusion_common::Result;
-use datafusion_expr::ScalarUDFImpl;
-use datafusion_expr::TypeSignature::Exact;
-use datafusion_expr::{ColumnarValue, Signature, Volatility};
-
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarUDFImpl, Signature, Volatility,
+};
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
#[derive(Debug)]
pub struct ContainsFunc {
@@ -44,22 +43,8 @@ impl Default for ContainsFunc {
impl ContainsFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::one_of(
- vec![
- Exact(vec![Utf8View, Utf8View]),
- Exact(vec![Utf8View, Utf8]),
- Exact(vec![Utf8View, LargeUtf8]),
- Exact(vec![Utf8, Utf8View]),
- Exact(vec![Utf8, Utf8]),
- Exact(vec![Utf8, LargeUtf8]),
- Exact(vec![LargeUtf8, Utf8View]),
- Exact(vec![LargeUtf8, Utf8]),
- Exact(vec![LargeUtf8, LargeUtf8]),
- ],
- Volatility::Immutable,
- ),
+ signature: Signature::string(2, Volatility::Immutable),
}
}
}
@@ -84,6 +69,37 @@ impl ScalarUDFImpl for ContainsFunc {
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
make_scalar_function(contains, vec![])(args)
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_contains_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_contains_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description(
+ "Return true if search_str is found within string (case-sensitive).",
+ )
+ .with_syntax_example("contains(str, search_str)")
+ .with_sql_example(
+ r#"```sql
+> select contains('the quick brown fox', 'row');
++---------------------------------------------------+
+| contains(Utf8("the quick brown fox"),Utf8("row")) |
++---------------------------------------------------+
+| true |
++---------------------------------------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_argument("search_str", "The string to search for in str.")
+ .build()
+ .unwrap()
+ })
}
/// use regexp_is_match_utf8_scalar to do the calculation for contains
@@ -92,41 +108,8 @@ pub fn contains(args: &[ArrayRef]) -> Result<ArrayRef, DataFusionError> {
(Utf8View, Utf8View) => {
let mod_str = args[0].as_string_view();
let match_str = args[1].as_string_view();
- let res = regexp_is_match_utf8::<
- StringViewArray,
+ let res = regexp_is_match::<
StringViewArray,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
- (Utf8View, Utf8) => {
- let mod_str = args[0].as_string_view();
- let match_str = args[1].as_string::<i32>();
- let res = regexp_is_match_utf8::<
- StringViewArray,
- GenericStringArray<i32>,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
- (Utf8View, LargeUtf8) => {
- let mod_str = args[0].as_string_view();
- let match_str = args[1].as_string::<i64>();
- let res = regexp_is_match_utf8::<
- StringViewArray,
- GenericStringArray<i64>,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
- (Utf8, Utf8View) => {
- let mod_str = args[0].as_string::<i32>();
- let match_str = args[1].as_string_view();
- let res = regexp_is_match_utf8::<
- GenericStringArray<i32>,
StringViewArray,
GenericStringArray<i32>,
>(mod_str, match_str, None)?;
@@ -136,7 +119,7 @@ pub fn contains(args: &[ArrayRef]) -> Result<ArrayRef, DataFusionError> {
(Utf8, Utf8) => {
let mod_str = args[0].as_string::<i32>();
let match_str = args[1].as_string::<i32>();
- let res = regexp_is_match_utf8::<
+ let res = regexp_is_match::<
GenericStringArray<i32>,
GenericStringArray<i32>,
GenericStringArray<i32>,
@@ -144,43 +127,10 @@ pub fn contains(args: &[ArrayRef]) -> Result<ArrayRef, DataFusionError> {
Ok(Arc::new(res) as ArrayRef)
}
- (Utf8, LargeUtf8) => {
- let mod_str = args[0].as_string::<i32>();
- let match_str = args[1].as_string::<i64>();
- let res = regexp_is_match_utf8::<
- GenericStringArray<i32>,
- GenericStringArray<i64>,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
- (LargeUtf8, Utf8View) => {
- let mod_str = args[0].as_string::<i64>();
- let match_str = args[1].as_string_view();
- let res = regexp_is_match_utf8::<
- GenericStringArray<i64>,
- StringViewArray,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
- (LargeUtf8, Utf8) => {
- let mod_str = args[0].as_string::<i64>();
- let match_str = args[1].as_string::<i32>();
- let res = regexp_is_match_utf8::<
- GenericStringArray<i64>,
- GenericStringArray<i32>,
- GenericStringArray<i32>,
- >(mod_str, match_str, None)?;
-
- Ok(Arc::new(res) as ArrayRef)
- }
(LargeUtf8, LargeUtf8) => {
let mod_str = args[0].as_string::<i64>();
let match_str = args[1].as_string::<i64>();
- let res = regexp_is_match_utf8::<
+ let res = regexp_is_match::<
GenericStringArray<i64>,
GenericStringArray<i64>,
GenericStringArray<i32>,
@@ -193,89 +143,3 @@ pub fn contains(args: &[ArrayRef]) -> Result<ArrayRef, DataFusionError> {
}
}
}
-
-#[cfg(test)]
-mod tests {
- use crate::string::contains::ContainsFunc;
- use crate::utils::test::test_function;
- use arrow::array::Array;
- use arrow::{array::BooleanArray, datatypes::DataType::Boolean};
- use datafusion_common::Result;
- use datafusion_common::ScalarValue;
- use datafusion_expr::ColumnarValue;
- use datafusion_expr::ScalarUDFImpl;
- #[test]
- fn test_functions() -> Result<()> {
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::from("alphabet")),
- ColumnarValue::from(ScalarValue::from("alph")),
- ],
- Ok(Some(true)),
- bool,
- Boolean,
- BooleanArray
- );
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::from("alphabet")),
- ColumnarValue::from(ScalarValue::from("dddddd")),
- ],
- Ok(Some(false)),
- bool,
- Boolean,
- BooleanArray
- );
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::from("alphabet")),
- ColumnarValue::from(ScalarValue::from("pha")),
- ],
- Ok(Some(true)),
- bool,
- Boolean,
- BooleanArray
- );
-
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::Utf8View(Some(String::from("Apache")))),
- ColumnarValue::from(ScalarValue::Utf8View(Some(String::from("pac")))),
- ],
- Ok(Some(true)),
- bool,
- Boolean,
- BooleanArray
- );
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::Utf8View(Some(String::from("Apache")))),
- ColumnarValue::from(ScalarValue::Utf8(Some(String::from("ap")))),
- ],
- Ok(Some(false)),
- bool,
- Boolean,
- BooleanArray
- );
- test_function!(
- ContainsFunc::new(),
- &[
- ColumnarValue::from(ScalarValue::Utf8View(Some(String::from("Apache")))),
- ColumnarValue::from(ScalarValue::LargeUtf8(Some(String::from(
- "DataFusion"
- )))),
- ],
- Ok(Some(false)),
- bool,
- Boolean,
- BooleanArray
- );
-
- Ok(())
- }
-}
diff --git a/datafusion/functions/src/string/ends_with.rs b/datafusion/functions/src/string/ends_with.rs
index 82fde772c2..42d673cf7b 100644
--- a/datafusion/functions/src/string/ends_with.rs
+++ b/datafusion/functions/src/string/ends_with.rs
@@ -16,18 +16,17 @@
// under the License.
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use arrow::array::ArrayRef;
use arrow::datatypes::DataType;
+use crate::utils::make_scalar_function;
use datafusion_common::{internal_err, Result};
-use datafusion_expr::TypeSignature::*;
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::utils::make_scalar_function;
-
#[derive(Debug)]
pub struct EndsWithFunc {
signature: Signature,
@@ -42,17 +41,7 @@ impl Default for EndsWithFunc {
impl EndsWithFunc {
pub fn new() -> Self {
Self {
- signature: Signature::one_of(
- vec![
- // Planner attempts coercion to the target type starting with the most preferred candidate.
- // For example, given input `(Utf8View, Utf8)`, it first tries coercing to `(Utf8View, Utf8View)`.
- // If that fails, it proceeds to `(Utf8, Utf8)`.
- Exact(vec![DataType::Utf8View, DataType::Utf8View]),
- Exact(vec![DataType::Utf8, DataType::Utf8]),
- Exact(vec![DataType::LargeUtf8, DataType::LargeUtf8]),
- ],
- Volatility::Immutable,
- ),
+ signature: Signature::string(2, Volatility::Immutable),
}
}
}
@@ -84,6 +73,41 @@ impl ScalarUDFImpl for EndsWithFunc {
}
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_ends_with_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_ends_with_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Tests if a string ends with a substring.")
+ .with_syntax_example("ends_with(str, substr)")
+ .with_sql_example(
+ r#"```sql
+> select ends_with('datafusion', 'soin');
++--------------------------------------------+
+| ends_with(Utf8("datafusion"),Utf8("soin")) |
++--------------------------------------------+
+| false |
++--------------------------------------------+
+> select ends_with('datafusion', 'sion');
++--------------------------------------------+
+| ends_with(Utf8("datafusion"),Utf8("sion")) |
++--------------------------------------------+
+| true |
++--------------------------------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_argument("substr", "Substring to test for.")
+ .build()
+ .unwrap()
+ })
}
/// Returns true if string ends with suffix.
diff --git a/datafusion/functions/src/string/initcap.rs b/datafusion/functions/src/string/initcap.rs
index 081fa94af8..cfac6f423f 100644
--- a/datafusion/functions/src/string/initcap.rs
+++ b/datafusion/functions/src/string/initcap.rs
@@ -16,18 +16,18 @@
// under the License.
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use arrow::array::{ArrayRef, GenericStringArray, OffsetSizeTrait, StringArray};
use arrow::datatypes::DataType;
+use crate::utils::{make_scalar_function, utf8_to_str_type};
use datafusion_common::cast::{as_generic_string_array, as_string_view_array};
use datafusion_common::{exec_err, Result};
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::utils::{make_scalar_function, utf8_to_str_type};
-
#[derive(Debug)]
pub struct InitcapFunc {
signature: Signature,
@@ -41,13 +41,8 @@ impl Default for InitcapFunc {
impl InitcapFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::uniform(
- 1,
- vec![Utf8, LargeUtf8, Utf8View],
- Volatility::Immutable,
- ),
+ signature: Signature::string(1, Volatility::Immutable),
}
}
}
@@ -79,6 +74,34 @@ impl ScalarUDFImpl for InitcapFunc {
}
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_initcap_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_initcap_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Capitalizes the first character in each word in the input string. Words are delimited by non-alphanumeric characters.")
+ .with_syntax_example("initcap(str)")
+ .with_sql_example(r#"```sql
+> select initcap('apache datafusion');
++------------------------------------+
+| initcap(Utf8("apache datafusion")) |
++------------------------------------+
+| Apache Datafusion |
++------------------------------------+
+```"#)
+ .with_standard_argument("str", "String")
+ .with_related_udf("lower")
+ .with_related_udf("upper")
+ .build()
+ .unwrap()
+ })
}
/// Converts the first letter of each word to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.
diff --git a/datafusion/functions/src/string/levenshtein.rs b/datafusion/functions/src/string/levenshtein.rs
index 430c402a50..558e71239f 100644
--- a/datafusion/functions/src/string/levenshtein.rs
+++ b/datafusion/functions/src/string/levenshtein.rs
@@ -16,7 +16,7 @@
// under the License.
use std::any::Any;
-use std::sync::Arc;
+use std::sync::{Arc, OnceLock};
use arrow::array::{ArrayRef, Int32Array, Int64Array, OffsetSizeTrait};
use arrow::datatypes::DataType;
@@ -25,8 +25,8 @@ use crate::utils::{make_scalar_function, utf8_to_int_type};
use datafusion_common::cast::{as_generic_string_array, as_string_view_array};
use datafusion_common::utils::datafusion_strsim;
use datafusion_common::{exec_err, Result};
-use datafusion_expr::ColumnarValue;
-use datafusion_expr::TypeSignature::*;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation};
use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
#[derive(Debug)]
@@ -43,14 +43,7 @@ impl Default for LevenshteinFunc {
impl LevenshteinFunc {
pub fn new() -> Self {
Self {
- signature: Signature::one_of(
- vec![
- Exact(vec![DataType::Utf8View, DataType::Utf8View]),
- Exact(vec![DataType::Utf8, DataType::Utf8]),
- Exact(vec![DataType::LargeUtf8, DataType::LargeUtf8]),
- ],
- Volatility::Immutable,
- ),
+ signature: Signature::string(2, Volatility::Immutable),
}
}
}
@@ -83,6 +76,33 @@ impl ScalarUDFImpl for LevenshteinFunc {
}
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_levenshtein_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_levenshtein_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Returns the [`Levenshtein distance`](https://en.wikipedia.org/wiki/Levenshtein_distance) between the two given strings.")
+ .with_syntax_example("levenshtein(str1, str2)")
+ .with_sql_example(r#"```sql
+> select levenshtein('kitten', 'sitting');
++---------------------------------------------+
+| levenshtein(Utf8("kitten"),Utf8("sitting")) |
++---------------------------------------------+
+| 3 |
++---------------------------------------------+
+```"#)
+ .with_argument("str1", "String expression to compute Levenshtein distance with str2.")
+ .with_argument("str2", "String expression to compute Levenshtein distance with str1.")
+ .build()
+ .unwrap()
+ })
}
///Returns the Levenshtein distance between the two given strings.
diff --git a/datafusion/functions/src/string/lower.rs b/datafusion/functions/src/string/lower.rs
index ca324e69c0..f82b11ca90 100644
--- a/datafusion/functions/src/string/lower.rs
+++ b/datafusion/functions/src/string/lower.rs
@@ -15,16 +15,16 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
use arrow::datatypes::DataType;
-
-use datafusion_common::Result;
-use datafusion_expr::ColumnarValue;
-use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+use std::any::Any;
+use std::sync::OnceLock;
use crate::string::common::to_lower;
use crate::utils::utf8_to_str_type;
+use datafusion_common::Result;
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
#[derive(Debug)]
pub struct LowerFunc {
@@ -39,13 +39,8 @@ impl Default for LowerFunc {
impl LowerFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::uniform(
- 1,
- vec![Utf8, LargeUtf8, Utf8View],
- Volatility::Immutable,
- ),
+ signature: Signature::string(1, Volatility::Immutable),
}
}
}
@@ -70,8 +65,37 @@ impl ScalarUDFImpl for LowerFunc {
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
to_lower(args, "lower")
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_lower_doc())
+ }
}
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_lower_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Converts a string to lower-case.")
+ .with_syntax_example("lower(str)")
+ .with_sql_example(
+ r#"```sql
+> select lower('Ångström');
++-------------------------+
+| lower(Utf8("Ångström")) |
++-------------------------+
+| ångström |
++-------------------------+
+```"#,
+ )
+ .with_standard_argument("str", "String")
+ .with_related_udf("initcap")
+ .with_related_udf("upper")
+ .build()
+ .unwrap()
+ })
+}
#[cfg(test)]
mod tests {
use super::*;
diff --git a/datafusion/functions/src/string/ltrim.rs b/datafusion/functions/src/string/ltrim.rs
index 0ddb5a205b..b64dcda721 100644
--- a/datafusion/functions/src/string/ltrim.rs
+++ b/datafusion/functions/src/string/ltrim.rs
@@ -15,20 +15,19 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
use arrow::array::{ArrayRef, OffsetSizeTrait};
use arrow::datatypes::DataType;
+use std::any::Any;
+use std::sync::OnceLock;
+use crate::string::common::*;
+use crate::utils::{make_scalar_function, utf8_to_str_type};
use datafusion_common::{exec_err, Result};
use datafusion_expr::function::Hint;
-use datafusion_expr::TypeSignature::*;
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, TypeSignature, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::string::common::*;
-use crate::utils::{make_scalar_function, utf8_to_str_type};
-
/// Returns the longest string with leading characters removed. If the characters are not specified, whitespace is removed.
/// ltrim('zzzytest', 'xyz') = 'test'
fn ltrim<T: OffsetSizeTrait>(args: &[ArrayRef]) -> Result<ArrayRef> {
@@ -49,18 +48,9 @@ impl Default for LtrimFunc {
impl LtrimFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
signature: Signature::one_of(
- vec![
- // Planner attempts coercion to the target type starting with the most preferred candidate.
- // For example, given input `(Utf8View, Utf8)`, it first tries coercing to `(Utf8View, Utf8View)`.
- // If that fails, it proceeds to `(Utf8, Utf8)`.
- Exact(vec![Utf8View, Utf8View]),
- Exact(vec![Utf8, Utf8]),
- Exact(vec![Utf8View]),
- Exact(vec![Utf8]),
- ],
+ vec![TypeSignature::String(2), TypeSignature::String(1)],
Volatility::Immutable,
),
}
@@ -104,6 +94,41 @@ impl ScalarUDFImpl for LtrimFunc {
),
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_ltrim_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_ltrim_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Trims the specified trim string from the beginning of a string. If no trim string is provided, all whitespace is removed from the start of the input string.")
+ .with_syntax_example("ltrim(str[, trim_str])")
+ .with_sql_example(r#"```sql
+> select ltrim(' datafusion ');
++-------------------------------+
+| ltrim(Utf8(" datafusion ")) |
++-------------------------------+
+| datafusion |
++-------------------------------+
+> select ltrim('___datafusion___', '_');
++-------------------------------------------+
+| ltrim(Utf8("___datafusion___"),Utf8("_")) |
++-------------------------------------------+
+| datafusion___ |
++-------------------------------------------+
+```"#)
+ .with_standard_argument("str", "String")
+ .with_argument("trim_str", "String expression to trim from the beginning of the input string. Can be a constant, column, or function, and any combination of arithmetic operators. _Default is whitespace characters._")
+ .with_related_udf("btrim")
+ .with_related_udf("rtrim")
+ .build()
+ .unwrap()
+ })
}
#[cfg(test)]
diff --git a/datafusion/functions/src/string/octet_length.rs b/datafusion/functions/src/string/octet_length.rs
index 93e62961bc..26ce70a416 100644
--- a/datafusion/functions/src/string/octet_length.rs
+++ b/datafusion/functions/src/string/octet_length.rs
@@ -15,17 +15,17 @@
// specific language governing permissions and limitations
// under the License.
-use std::any::Any;
-
use arrow::compute::kernels::length::length;
use arrow::datatypes::DataType;
+use std::any::Any;
+use std::sync::OnceLock;
+use crate::utils::utf8_to_int_type;
use datafusion_common::{exec_err, Result, ScalarValue};
-use datafusion_expr::{ColumnarValue, Volatility};
+use datafusion_expr::scalar_doc_sections::DOC_SECTION_STRING;
+use datafusion_expr::{ColumnarValue, Documentation, Volatility};
use datafusion_expr::{ScalarUDFImpl, Signature};
-use crate::utils::utf8_to_int_type;
-
#[derive(Debug)]
pub struct OctetLengthFunc {
signature: Signature,
@@ -39,13 +39,8 @@ impl Default for OctetLengthFunc {
impl OctetLengthFunc {
pub fn new() -> Self {
- use DataType::*;
Self {
- signature: Signature::uniform(
- 1,
- vec![Utf8, LargeUtf8, Utf8View],
- Volatility::Immutable,
- ),
+ signature: Signature::string(1, Volatility::Immutable),
}
}
}
@@ -91,6 +86,36 @@ impl ScalarUDFImpl for OctetLengthFunc {
},
}
}
+
+ fn documentation(&self) -> Option<&Documentation> {
+ Some(get_octet_length_doc())
+ }
+}
+
+static DOCUMENTATION: OnceLock<Documentation> = OnceLock::new();
+
+fn get_octet_length_doc() -> &'static Documentation {
+ DOCUMENTATION.get_or_init(|| {
+ Documentation::builder()
+ .with_doc_section(DOC_SECTION_STRING)
+ .with_description("Returns the length of a string in bytes.")
+ .with_syntax_example("octet_length(str)")
+ .with_sql_example(
... 10885 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org