You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "caseykneale (via GitHub)" <gi...@apache.org> on 2023/08/24 11:45:03 UTC
[GitHub] [arrow-datafusion] caseykneale opened a new issue, #7394: Regularly Segfaulting in Nested Query
caseykneale opened a new issue, #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394
### Describe the bug
The data I am using is in a normalized parquet format and each "table" is in its own file, each registered to the SessionContext instance, and can be queried with a variety of other queries. The problematic query has maybe 4 `JOIN`s (outer, inner, etc) and it ends with a `NOT EXISTS` clause on a subquery with another set of the roughly the same 4 `JOIN`s on different tables. The outer join occurs inside of a view which is shared by the other 3 inner joins. The query planning succeeds. The query runs for a while, maybe 15 minutes, and then appears as though it has completed (CPU cores spin down, RAM consumption goes down to baseline). The SegFault happens during the collection of the DataFrame itself(it's `await`ed on) before the RecordBatches are collected from the Dataframe. For what its worth, the dataframe should be empty at the end of this query as its serving as a control for a unit test.
Then segfault currently occurs on an intel Mac. I saw an open issue about seg faulting in unit tests https://github.com/apache/arrow-datafusion/issues/5693 and don't know whether or not this could be the same issue.
I see a few blocks of unsafe code in the project, most of which look benign, but I haven't ruled out a stack overflow scenario. Not sure where to poke at. May try adjusting `RUST_MIN_STACK` to see if that helps? Or memoizing the subquery results before the `NOT EXIST` call?
Any suggestions appreciated.
### To Reproduce
I can't share the data to reproduce this or the code unfortunately but something tells me I could make a MRE as I doubt this behavior is exclusive to this type of query.
### Expected behavior
This may sound terse but I mean this in the most polite way possible. Ideally queries do not segfault.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Regularly Segfaulting in NOT EXIST query [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7394: Regularly Segfaulting in NOT EXIST query
URL: https://github.com/apache/datafusion/issues/7394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] caseykneale commented on issue #7394: Regularly Segfaulting in NOT EXIST query
Posted by "caseykneale (via GitHub)" <gi...@apache.org>.
caseykneale commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1693574796
I was able to work around these issues by sorting the data frames and single thread walking the OUTER JOIN/NOT EXISTS in rust without getting anything near an OOM, nor a SO. This is at least 15x faster than the failed runs despite costs to serialization, etc. I recommend this approach as a workaround should anyone else run into this.
I do hope to see the performance and correctness of these queries improved because I really do like this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] caseykneale commented on issue #7394: Regularly Segfaulting in Nested Query
Posted by "caseykneale (via GitHub)" <gi...@apache.org>.
caseykneale commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1691540450
Yes it is running in release. I do `cargo build --release` then run the binary. I can share the output, but it basically has a println after the query plan occurs and then a segfault which occurs before a println saying the dataframe/query was collected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on issue #7394: Regularly Segfaulting in NOT EXIST query
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1722296323
I think we should understand what is going on here. Can you post some dummy data and an accompanying query to reproduce on our end?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #7394: Regularly Segfaulting in Nested Query
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1691537446
Just a check, is your project running in release mode? `cargo run --release`? Can you share the output / example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Regularly Segfaulting in NOT EXIST query [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7394:
URL: https://github.com/apache/datafusion/issues/7394#issuecomment-2836530403
This hasn't had any update for 1.5 years, so closing it to clean up our backlog. It doesn't seem actionable to me. Please reopen if there is any update or you are able to provide example data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] caseykneale commented on issue #7394: Regularly Segfaulting in NOT EXIST query
Posted by "caseykneale (via GitHub)" <gi...@apache.org>.
caseykneale commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1692183835
if I replace the AND's with checks for IS NOT NULL first on the optional fields, I now OOM. The OOM is surprising, the data isn't that large (<500 MB on disk uncompressed) and the query itself isn't that demanding but I'm blowing through 50GB SWAP and 16GB RAM.
So it seems like I am possibly blowing out the stack or there is a memory bug associated with NOT EXISTS with WHERE clauses that compare NULL's. But in general the memory consumption here seems quite high for a query returning zero records.
I might do better JOINing the two tables and looking for NULLs in a for loop post serialization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] caseykneale commented on issue #7394: Regularly Segfaulting in Nested Query
Posted by "caseykneale (via GitHub)" <gi...@apache.org>.
caseykneale commented on issue #7394:
URL: https://github.com/apache/arrow-datafusion/issues/7394#issuecomment-1691990787
I have distilled the query down specifically to a `NOT EXIST` OR a `LEFT JOIN` by creating in memory tables of the join queries. My segfaulting query looks like this:
```SQL
SELECT
lhs.primary_key,
lhs.uuid,
lhs.thing,
lhs.other_thing,
lhs.other_other_thing
FROM lhs
WHERE NOT EXISTS (
SELECT * FROM rhs
WHERE lhs.uuid = rhs.uuid
AND lhs.thing = rhs.thing
AND lhs.other_thing = rhs.other_thing
AND lhs.other_other_thing = rhs.other_other_thing
)
```
where all types are nullable strings, or int64s.
At first I thought it had to do with nesting, now I am wondering if it has to do with nullable comparisons inside of a not exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Regularly Segfaulting in NOT EXIST query [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7394: Regularly Segfaulting in NOT EXIST query
URL: https://github.com/apache/datafusion/issues/7394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org