You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2025/02/07 04:42:08 UTC
[I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
2010YOUY01 opened a new issue, #14535:
URL: https://github.com/apache/datafusion/issues/14535
### Is your feature request related to a problem or challenge?
This a project idea for GSoC 2025 https://github.com/apache/datafusion/issues/14478
`datafusion-sqlancer` is a SQL level fuzz testing implementation for DataFusion. https://github.com/apache/datafusion/issues/11030
## Current implementation status
`datafusion-sqlancer` has covered partial SQL features, and data types, and implemented 3 relatively simple testing oracles[^1]. With occasional manual runs, around 50 bugs have been found.
The implementation is in Java, and it's a fork of the original [SQLancer](https://github.com/sqlancer/sqlancer).
## Why rewrite in Rust
The SQLancer was first implemented in Java for very good reasons: it has to test the effectiveness of several testing oracles on many major databases, JDBC is a common interface.
DataFusion's SQLancer implementation now is done by extending SQLancer framework, it has saved us some effort to do CLI parsing, result comparison, etc.
There are several reasons I think it's a good idea to rewrite in Rust at this point:
- (major) **Making test oracles also apply to `sqllogictests`**
`datafusion-sqlancer` consists of two modules: random query generation, and property validation for test oracles. Those properties can also be applied to enhance existing [SQL tests](https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest). If we have those properties implemented in Rust, enhancing existing `sqllogictest`s would be easier.
Now only 3 simple test oracles have been implemented, and I believe there are around 10 novel SQL testing algorithms have been proposed, one example is `Equivalent Expression Transformation`(https://www.usenix.org/conference/osdi24/presentation/jiang). EET I think is very suitable to enhance existing SQL tests.
Overall, I think it's a good time to switch to native rust implementation before implementing more complex testing algorithms.
- **Simplier implementation**
One thing we simplify is now we don't have to use JDBC to connect the testing framework and DataFusion core, configuration fuzzing can be easier, and there might be some existing code we can reuse.
- **More contributors**
DataFusion ecosystem is mainly in Rust, IMO it would be easier to find people to help if the testing framework is written in Rust instead of Java.
[^1]: https://github.com/apache/datafusion/issues/11030 has a minimal example for testing oracle `NoREC`
### Describe the solution you'd like
See https://github.com/apache/datafusion/issues/11030 for the background
- Generate random query to a datafusion internal data structure (perhaps `Statement`)
- Implement testing oracles. In order to support also running with existing SQL tests, we might want:
- For query mutation: mutate the query's internal representation, and convert it back to SQL string
- For property check: implement by extending `sqllogictest` framework
### Describe alternatives you've considered
The project idea proposed above I believe is advanced in terms of difficulty.
A medium level project can be extending existing implementation with more SQL/types support, and implement more test oracles, also with better CI integration.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2834586678
> > I bet we are not the only project that would like to have SQLLancer type support in Rust
>
> That's really cool! Databend will definitely be interesting in building this project. So does iceberg-rust.
Hi @Xuanwo 😄 , how's it going?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "andkolbe (via GitHub)" <gi...@apache.org>.
andkolbe commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2716333456
Thank you very much for your quick and thorough response. I'll keep digging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "arpity22 (via GitHub)" <gi...@apache.org>.
arpity22 commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-3245371095
Cool! I’ll stick around and keep reading up on things. Feel free to ping me if you think I can help, or create an issue on your repo, I’ll check upon them once in a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2643713623
> I bet we are not the only project that would like to have SQLLancer type support in Rust
That's really cool! Databend will definitely be interesting in building this project. So does iceberg-rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2712646798
> Hello, I am interested in applying to work on this project for GSoC. After reading through [#11030](https://github.com/apache/datafusion/issues/11030) , it looks like the three testing oracles that have been implemented from SQLancer are NoREC, TLP, and PQS. Were those chosen because they were the easiest to implement, or was there something about how they test Datafusion specifically?
👋🏼 They're implemented first because
- All of them are general-purpose algorithms, which should work in most systems:
- Taking TLP for example, it's targeting edge case value handling (NULLs) so it should work well for DataFusion.
- NoREC: It's checking the consistency between optimized (by predicate pushdown) path and non-optimized path, and also it's checking the consistency between how a same predicate is evaluated in `select expr` and `where expr`, it also caught several bugs for DataFusion
- PQS: I don't have very good intuition on why this one should work and I think it has caught 0 or 1 bug 🤦🏼 Perhaps I'm missing something.
- Yes, they're also very easy to implement.
To make fuzzing more specific to DataFusion, I think the most needed is configuration fuzzing or data source fuzzing. To make `DataFusion` more performant. For the same executor there are many specialized execution paths, and they're controlled by turning different configuration knobs in https://datafusion.apache.org/user-guide/configs.html
However, they're quite hard to implement due to the complexity. Given a randomly generated query, if we pick a random configuration for every option, it's very likely to fail, becuase this configuration is invalid, I believe given a specific query only a small subset of the configurations are relevant. So now we implement this kind of configuration fuzzing separately, for example https://github.com/apache/datafusion/blob/main/datafusion/core/tests/fuzz_cases/aggregation_fuzzer/mod.rs
I wish configuration fuzzing can be integrated into `datafusion-sqlancer`, but it still has a long way to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "andkolbe (via GitHub)" <gi...@apache.org>.
andkolbe commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2711744502
Hello, I am interested in applying to work on this project for GSoC. After reading through #11030 , it looks like the three testing oracles that have been implemented from SQLancer are NoREC, TLP, and PQS. Were those chosen because they were the easiest to implement, or was there something about how they test Datafusion specifically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-3245015061
> Hi! This looks really interesting. I’d like to work on it (just as a side project for fun). Is anyone else working on it at the moment who I could sync up with?
I'm working on it right now. I'll post an update here once the major components are finished (hopefully in 1–2 months). After the major components are stable, the project will be open to contributions. There will still be a lot of work needed to improve the test coverage.
The repo is at https://github.com/2010YOUY01/datafusion-fuzzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "arpity22 (via GitHub)" <gi...@apache.org>.
arpity22 commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-3243736427
Hi! This looks really interesting. I’d like to work on it (just as a side project for fun). Is anyone else working on it at the moment who I could sync up with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14535:
URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2643689301
I bet we are not the only project that would like to have SQLLancer type support in Rust
Perhaps we can join forces (aka with Risingwave)
ps. @Xuanwo -- here is another example of "rewrite it in Rust!"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org