You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "thinkharderdev (via GitHub)" <gi...@apache.org> on 2023/03/22 14:08:52 UTC
[GitHub] [arrow-datafusion] thinkharderdev opened a new issue, #5693: Logical Expr serde should be stack safe
thinkharderdev opened a new issue, #5693:
URL: https://github.com/apache/arrow-datafusion/issues/5693
### Is your feature request related to a problem or challenge?
Noticed while working on https://github.com/apache/arrow-datafusion/issues/5691 I was getting stack overflows while running tests (well actually a segfault but after some investigation I think it was just a quirk of my ARM mac and really the underlying issue was a stack overflow). I was able to run the tests by compiling them in release mode but the current recursive expression parsing/translation is not stack safe and will likely eventually cause issues.
Otherwise deeply nested expressions can cause a system crash.
### Describe the solution you'd like
Avoid recursion in `datafusion_proto::logical_plan::from_proto::parse_expr`
### Describe alternatives you've considered
Leave it as is and document as a known limitation
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5693: Logical Expr serde should be stack safe
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5693:
URL: https://github.com/apache/arrow-datafusion/issues/5693#issuecomment-1615144311
I think it changing to use a worklist style algorithm (store state on the heap rather than the stack with recursive function calls) would be neat, though it may be a big change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jonahgao commented on issue #5693: Logical Expr serde should be stack safe
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on issue #5693:
URL: https://github.com/apache/arrow-datafusion/issues/5693#issuecomment-1612512080
Hi, @thinkharderdev
I might have encountered the same problem.
Could you please try to increase the stack size here: https://github.com/apache/arrow-datafusion/blob/06e22a5386a65c0e6a29943cd03ab5e442732745/datafusion/proto/src/bytes/mod.rs#L458
I increased it to `12_000_000`, and problem disappeared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on issue #5693: Logical Expr serde should be stack safe
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #5693:
URL: https://github.com/apache/arrow-datafusion/issues/5693#issuecomment-1481609693
thats cool @thinkharderdev, is it possible to get a valid reproduce scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org