You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/12/30 13:36:22 UTC

[I] Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test [datafusion]

alamb opened a new issue, #13946:
URL: https://github.com/apache/datafusion/issues/13946

   After https://github.com/apache/datafusion/pull/12978 the `test_fuzz_utf8` test takes over a minute to run on my machine
   
   This ticket tracks improving the speed
   
                 > Thanks @adriangb -- I think this PR is ready to go
   > 
   > One thing I noticed is that the fuzz test takes over a minute on my machine:
   > 
   > ```
   >         SLOW [> 60.000s] datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8
   >         PASS [  65.772s] datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8
   > ------------
   >      Summary [  72.749s] 47 tests run: 47 passed (1 slow), 0 skipped
   > andrewlamb@Mac:~/Software/datafusion$
   > ```
   > 
   > Is there some way to make it faster? Maybe with multiple threads or crank down the number of things to teset?
   
   Yeah this is what I was hinting at in https://github.com/apache/datafusion/pull/12978#issuecomment-2542335627.
   
   I'm happy to throw threads at it for a start, and restricting the search space might be necessary but I think requires a more careful eye to minimize how much valuable testing is discarded. The other thing that I think we can do is speed up the tests themselves, in particular minimizing unnecessary round trips to Parquet, but I'm not sure where the right places to hook in would be that still give us a realistic test but remove the need to re-parse the same data over and over again.
   
   _Originally posted by @adriangb in https://github.com/apache/datafusion/issues/12978#issuecomment-2563737914_
               


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13946:
URL: https://github.com/apache/datafusion/issues/13946#issuecomment-2565575247

   - Proposal: https://github.com/apache/datafusion/pull/13947#top


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13946:
URL: https://github.com/apache/datafusion/issues/13946#issuecomment-2565510581

   I am going to give this a try


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13946: Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test
URL: https://github.com/apache/datafusion/issues/13946


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Improve speed of `datafusion::fuzz fuzz_cases::pruning::test_fuzz_utf8` test [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #13946:
URL: https://github.com/apache/datafusion/issues/13946#issuecomment-2565546423

   Amazing thanks Andrew!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org