[GitHub] [calcite] zabetak commented on pull request #3145: [CALCITE-5615] Program to run SQL Logic Tests for Calcite

["zabetak (via GitHub)" <gi...@apache.org>]

zabetak commented on PR #3145:
URL: https://github.com/apache/calcite/pull/3145#issuecomment-1580300860

   > It took about 6 minutes to run 10K tests using CalciteExecutor, so by extrapolation it would take about 2 days to run all tests.
   
   Here are some thoughts regarding duration and frequency:
   * Tests runtime ~5 minutes -> could run on every PR;
   * Tests runtime between 1h and 3h -> could possibly run once daily;
   * Tests runtime under 6h -> could run once per week;
   * Tests runtime spanning days -> per request/ not automated (maybe before release)
   It's up to us I guess to divide them into meaningful buckets (e..g, those that are likely to break more often should run on every PR) and setup the relevant CI infra.
   
   Obviously we don't have to setup everything as part of this PR but it would be nice if we could at least put the 5minute bucket in place.
   
   > Memory consumption increases slightly with time.
   
   I guess we can find ways to address memory pressure; some quick ideas:
   * Run each test file separately; this in principle should reset the memory after each run.
   * Provide options/APIs to save failures in files (instead of keeping ever growing in memory structures)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org