[GitHub] [druid] paul-rogers commented on pull request #12992: Add IT for MSQ task engine using the new IT framework

[GitBox <gi...@apache.org>]

paul-rogers commented on PR #12992:
URL: https://github.com/apache/druid/pull/12992#issuecomment-1249987215

   An earlier comment mentioned that it would be handy to use the file format from the oft-discussed, but not yet committed planner test framework. To make this more concrete, I just opened a [PR #13106](https://github.com/apache/druid/pull/13106) with just the test file format, without the unit test mechanisms. This provides the option, in this PR, to use the proposed file format. The query file format is designed to validate the planer and so has many sections. Only a few are relevant to this PR. The thought is it might be simpler to reuse a subset of the test framework than it would be to reinvent the bits needed for an IT.
   
   A typical test case might look like this:
   
   ```
   === case
   An example insert test
   === sql
   INSERT INTO myTable
   SELECT ...
   ```
   
   A `SELECT` query can include JSON-formatted results.
   
   With this file format, a test would:
   
   * Load the test file
   * Loop over each test case
   * Grab the SQL and execute it. If there is a need, you can also grab the query context.
   * For an INSERT, just make sure the statement succeeds.
   * For a SELECT, verify that the actual results match the expected results, and that the schema matches the expected schema.
   
   The test framework PR includes an "actual results" class, but that might be overkill for the needs of an IT, since the IT will check only one or two expected results.
   
   The mechanism does have code to verify results and display errors. When run in a UT, the framework produces an "actual" output file using the same format as the "expected" file. This makes it easy to use a diff tool to see errors and to capture new expected values. That won't work in an IT context, but we can devise something simpler, maybe just showing the actual results and the line at which the results differ. (The test code will provide an error with the line number.)
   
   Perhaps there is a need to validate the reports. This can be done by adding a new section which would probably use the regex feature that turns a line into a regex if the line starts with "!". Also, "**" matches any number of lines:
   
   ```text
   === reports
   {
     "status": "SUCCESS",
   !  "runTimeMs": \d+,
   **
   ...
   ```
   
   Anyway, take a look. The code can be a bit daunting on first inspection, but the interface should be pretty simple. See the unit tests for examples.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org