You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "blaginin (via GitHub)" <gi...@apache.org> on 2025/06/02 16:50:44 UTC
[I] `sort_fuzz` testing DX improvements [datafusion]
blaginin opened a new issue, #16233:
URL: https://github.com/apache/datafusion/issues/16233
### Is your feature request related to a problem or challenge?
`datafusion/core/tests/fuzz_cases` is a very cool too! It helped to spot https://github.com/apache/datafusion/issues/16228. However, once the issue was discovered, it tool several hours to understand a problem and create a reproducible example
Partially that's because debug info to reproduce the problem right now is very big and not very useful:
<img width="1297" alt="Image" src="https://github.com/user-attachments/assets/e78a76dd-7daa-4604-a6be-3621d94bbc3b" />
### Describe the solution you'd like
Ideas I have to improve the DX:
- [ ] Store dataset in a file instead of printing to stdout. The current version is too big and a bit messy - and it also makes reproducing issues harder.
- [ ] Streamline seeds. There are some hardcoded seeds, but it's unclear how to call a function with arguments to get the same error-inducing dataset. It'd help to include a hint on how to generate a test batch for debugging later.
- [ ] Add docs on how to extend, use, and maintain the suite (including the modules above).
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `sort_fuzz` testing DX improvements [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #16233:
URL: https://github.com/apache/datafusion/issues/16233#issuecomment-2933126311
I agree with this, I've encountered some of these, especially this:
> Streamline seeds. There are some hardcoded seeds, but it's unclear how to call a function with arguments to get the same error-inducing dataset. It'd help to include a hint on how to generate a test batch for debugging
I'm thinking the plan would be to:
1. During failure we either set a environment variable to the seed and have it rerun, or print it out (or both)
2. Reproduce the dataset with the saved seed + save it to an arrow file
Something like (example where seed = 1 failed):
``` FUZZ_SEED=1 cargo test fuzz_case_example```
3. Have a reader for the arrow file
4. Add documentation to show how to do this in `fuzz_cases`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `sort_fuzz` testing DX improvements [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on issue #16233:
URL: https://github.com/apache/datafusion/issues/16233#issuecomment-2940217723
hey @jonathanc-n
> set an environment variable to the seed
I think it's a good idea to print the seed used, but not sure we should implicitly set the env variable.
One challenge with reproducing via seeds as currently implemented is that a single seed produces a [group of fuzzy tasks](https://github.com/apache/datafusion/blob/f513e2c0c22aa4f89d232da4bb01330ac4b02bce/datafusion/core/tests/fuzz_cases/aggregation_fuzzer/fuzzer.rs#L180), so it's unclear how to reference one specific task. Maybe dumping to a parquet or arrow_ipc file is the only simple way to make reproduction simple...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `sort_fuzz` testing DX improvements [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #16233:
URL: https://github.com/apache/datafusion/issues/16233#issuecomment-2933128589
@2010YOUY01 I think you'd be interested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org