You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nathanb9 (via GitHub)" <gi...@apache.org> on 2026/06/13 18:05:18 UTC
[PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
nathanb9 opened a new pull request, #22943:
URL: https://github.com/apache/datafusion/pull/22943
## Which issue does this PR close?
- Closes #22936.
## Rationale for this change
A flat `WHERE col = 0 OR col = 1 OR ... ` chain builds a very deep `BinaryExpr` tree. The tree is built iteratively so building it is fine, but later recursive passes (derived `Expr::clone`, `Drop`, the optimizer) overflow the stack and abort the process. sqlparser's `recursion_limit` only guards *nested* expressions; flat equal-precedence chains are parsed iteratively and bypass it. We should fail gracefully with a planning error instead of crashing.
## What changes are included in this PR?
- Track expression-tree depth in the `sql_expr_to_logical_expr` stack machine, accumulating across nested sub-expressions via `PlannerContext`, and return a planning error when it exceeds `recursion_limit * 10`.
## Are these changes tested?
Yes. New unit tests cover: chains within the limit succeed, over-deep chains return a graceful error, and depth accumulates across nesting. Existing deep-chain tests still pass.
## Are there any user-facing changes?
Pathologically deep expressions now return a planning error instead of crashing. No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#discussion_r3453218001
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -54,6 +54,13 @@ mod substring;
mod unary_op;
mod value;
+/// Multiplier applied to `datafusion.sql_parser.recursion_limit` to bound the
+/// depth of a binary-operator expression tree. A flat operator chain such as
+/// `a OR b OR c OR ...` only consumes a single level of parser recursion yet
Review Comment:
is there a reason that a `a OR b OR c OR ..` only consumes a single parser recursion level?
Would it be possible to just update the code so that `a OR b OR c ...` updates the recursion protection correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#issuecomment-4859105436
Closing this. New issue tracks the broader problem https://github.com/apache/datafusion/issues/23264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#discussion_r3478436349
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -54,6 +54,13 @@ mod substring;
mod unary_op;
mod value;
+/// Multiplier applied to `datafusion.sql_parser.recursion_limit` to bound the
+/// depth of a binary-operator expression tree. A flat operator chain such as
+/// `a OR b OR c OR ...` only consumes a single level of parser recursion yet
Review Comment:
Or we expect this to be checked from other inputs as well? I think that might make more sense actually
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -54,6 +54,13 @@ mod substring;
mod unary_op;
mod value;
+/// Multiplier applied to `datafusion.sql_parser.recursion_limit` to bound the
+/// depth of a binary-operator expression tree. A flat operator chain such as
+/// `a OR b OR c OR ...` only consumes a single level of parser recursion yet
Review Comment:
Or we expect this to already be checked from other inputs as well? I think that might make more sense actually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#issuecomment-4847274822
Does your problem still happen if you enable the `recursive_protection` feature flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#discussion_r3426542847
##########
datafusion/sql/src/planner.rs:
##########
@@ -276,6 +276,13 @@ pub struct PlannerContext {
set_expr_left_schema: Option<DFSchemaRef>,
/// The parameters of all lambdas seen so far
lambda_parameters: HashMap<String, FieldRef>,
+ /// Depth of the SQL expression currently being planned, relative to the
Review Comment:
Nice catch carrying `expr_depth` across cloned query and subquery contexts. The `# Cloning` docs above still mention that only `ctes` are truly cloned. Could we update that comment to reflect the current behavior as well? That should help avoid future changes accidentally resetting or special-casing `expr_depth` during context cloning.
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -1494,6 +1552,71 @@ mod tests {
test_stack_overflow!(test_stack_overflow_2048, 2048);
test_stack_overflow!(test_stack_overflow_4096, 4096);
test_stack_overflow!(test_stack_overflow_8192, 8192);
+
+ /// Plan `sql` as an expression against an empty schema using the default
+ /// configuration (so the default expression-depth limit applies).
+ fn plan_expr_with_default_limit(sql: &str) -> Result<Expr> {
+ let schema = DFSchema::empty();
+ let mut planner_context = PlannerContext::default();
+
+ let dialect = GenericDialect {};
+ let mut parser = Parser::new(&dialect).try_with_sql(sql).unwrap();
+ let sql_expr = parser.parse_expr().unwrap();
+
+ let context_provider = TestContextProvider::new();
+ let sql_to_rel = SqlToRel::new(&context_provider);
+ sql_to_rel.sql_expr_to_logical_expr(sql_expr, &schema, &mut planner_context)
+ }
+
+ // The default expression-depth limit is `recursion_limit * 10`.
+ const DEFAULT_EXPR_DEPTH_LIMIT: usize = 50 * EXPR_DEPTH_LIMIT_MULTIPLIER;
+
+ #[test]
+ fn test_expr_depth_limit_within_limit() {
+ // A flat chain of N predicates produces a tree of depth N + 1, so the
+ // deepest chain accepted by the default limit has
+ // `DEFAULT_EXPR_DEPTH_LIMIT - 1` predicates.
+ plan_or_chain(DEFAULT_EXPR_DEPTH_LIMIT - 1).unwrap();
+ }
+
+ #[test]
+ fn test_expr_depth_limit_exceeded_returns_error() {
Review Comment:
The new regression tests cover `sql_expr_to_logical_expr` directly, which is great. As a follow-up, it might be worth adding a small end-to-end SQL planning test too. That would help demonstrate that the public `SessionContext`/`SessionState` path surfaces this planning error before validation or optimizer cloning has a chance to recurse into the deeply nested `Expr`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#issuecomment-4856671063
>Does your problem still happen if you enable the recursive_protection feature flag?
This specific issue (#22936) stack overflows is in derived Clone so not possible...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 commented on code in PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#discussion_r3478433513
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -54,6 +54,13 @@ mod substring;
mod unary_op;
mod value;
+/// Multiplier applied to `datafusion.sql_parser.recursion_limit` to bound the
+/// depth of a binary-operator expression tree. A flat operator chain such as
+/// `a OR b OR c OR ...` only consumes a single level of parser recursion yet
Review Comment:
Back from vacation!
You're saying can we update the code to have recursion protection in parser level? That makes sense, ill make a separate issue for this. But I think we still need this check here too.
There could still be an issue if its not a SQL input (DataFrame/Substrait/programmatic). sqlparser can be updated to have the recursion protection but couldnt other inputs cause this same issue if not checked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22943:
URL: https://github.com/apache/datafusion/pull/22943#discussion_r3492612878
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -54,6 +54,13 @@ mod substring;
mod unary_op;
mod value;
+/// Multiplier applied to `datafusion.sql_parser.recursion_limit` to bound the
+/// depth of a binary-operator expression tree. A flat operator chain such as
+/// `a OR b OR c OR ...` only consumes a single level of parser recursion yet
Review Comment:
👋
What i was saying is that I think there are already several places that check recursion depth and that we should figure out why those existing protections aren't working in this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: graceful error for deeply nested expressions instead of stack overflow [datafusion]
Posted by "nathanb9 (via GitHub)" <gi...@apache.org>.
nathanb9 closed pull request #22943: fix: graceful error for deeply nested expressions instead of stack overflow
URL: https://github.com/apache/datafusion/pull/22943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org