You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@calcite.apache.org by "Julian Hyde (Jira)" <ji...@apache.org> on 2023/02/15 22:56:00 UTC
[jira] [Updated] (CALCITE-5529) Improve dialect tests
[ https://issues.apache.org/jira/browse/CALCITE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julian Hyde updated CALCITE-5529:
---------------------------------
Description:
The current dialect tests are quick and robust but not very thorough. In {{RelToSqlConverterTest}} there are about 400 tests (374 SQL statements and 30 relational expressions) that test translation to 538 SQL strings in target dialects (an average of 1.34 dialects per test). To run the 371 test methods takes about 12 seconds elapsed, 80 CPU seconds on my MacBook Pro. None of those SQL strings are executed, so we don't know whether they return the correct results or are even valid.
Improvements:
* Define a list of dialects that we wish to test (the "target dialects") and provide a means to connect to a database for each of these (typically a driver on the class path, a URL, user name and password);
* In each test, for each target dialect, generate a SQL string. The generation should not throw.
* In each test, the query is first executed against Calcite to create a sample. (For any test, this can be overridden to use a different reference dialect. Calcite will use a recording, if present.)
* In each test, for each target dialect that has an *enabled* connection, execute the generated SQL against that connection. If there is a reference result for the test, check that the result equals the reference result.
* In each test, for each target dialect that has a *recording* connection, execute the generated SQL against that connection. Record the outcome (error, column names and types, result) in the file. Proceed as for an enabled connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying+recording* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.) If the outcome is not present, proceed as for a recording connection. (Execute the SQL, record the outcome, check against reference result, etc.)
With these improvements, the test suite:
* Remains fast and robust (because recordings are used most of the time);
* Can be run offline (using recordings);
* Can be run by contributors who do not necessarily have access to all of the supported databases;
* Has full coverage (runs all tests against all supported dialects);
* Can re-generate the 'golden' references. (Must be online, have the required driver and connection, and takes more time.)
Design:
* A new {{class DialectTestConfig}} describes the dialects to be tested, their names, connections, the connection state (enabled, recording, replaying, replaying+recording), and recording file name.
* A mock JDBC driver mediates recording/replay.
was:
The current dialect tests are quick and robust but not very thorough. In {{RelToSqlConverterTest}} there are about 400 tests (374 SQL statements and 30 relational expressions) that test translation to 538 SQL strings in target dialects (an average of 1.34 dialects per test). None of those SQL strings are executed, so we don't know whether they return the correct results or are even valid.
Improvements:
* Define a list of dialects that we wish to test (the "target dialects") and provide a means to connect to a database for each of these (typically a driver on the class path, a URL, user name and password);
* In each test, for each target dialect, generate a SQL string. The generation should not throw.
* In each test, the query is first executed against Calcite to create a sample. (For any test, this can be overridden to use a different reference dialect. Calcite will use a recording, if present.)
* In each test, for each target dialect that has an *enabled* connection, execute the generated SQL against that connection. If there is a reference result for the test, check that the result equals the reference result.
* In each test, for each target dialect that has a *recording* connection, execute the generated SQL against that connection. Record the outcome (error, column names and types, result) in the file. Proceed as for an enabled connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.)
* In each test, for each target dialect that has a *replaying+recording* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.) If the outcome is not present, proceed as for a recording connection. (Execute the SQL, record the outcome, check against reference result, etc.)
With these improvements, the test suite:
* Remains fast and robust (because recordings are used most of the time);
* Can be run offline (using recordings);
* Can be run by contributors who do not necessarily have access to all of the supported databases;
* Has full coverage (runs all tests against all supported dialects);
* Can re-generate the 'golden' references. (Must be online, have the required driver and connection, and takes more time.)
Design:
* A new {{class DialectTestConfig}} describes the dialects to be tested, their names, connections, the connection state (enabled, recording, replaying, replaying+recording), and recording file name.
* A mock JDBC driver mediates recording/replay.
> Improve dialect tests
> ---------------------
>
> Key: CALCITE-5529
> URL: https://issues.apache.org/jira/browse/CALCITE-5529
> Project: Calcite
> Issue Type: Bug
> Reporter: Julian Hyde
> Priority: Major
>
> The current dialect tests are quick and robust but not very thorough. In {{RelToSqlConverterTest}} there are about 400 tests (374 SQL statements and 30 relational expressions) that test translation to 538 SQL strings in target dialects (an average of 1.34 dialects per test). To run the 371 test methods takes about 12 seconds elapsed, 80 CPU seconds on my MacBook Pro. None of those SQL strings are executed, so we don't know whether they return the correct results or are even valid.
> Improvements:
> * Define a list of dialects that we wish to test (the "target dialects") and provide a means to connect to a database for each of these (typically a driver on the class path, a URL, user name and password);
> * In each test, for each target dialect, generate a SQL string. The generation should not throw.
> * In each test, the query is first executed against Calcite to create a sample. (For any test, this can be overridden to use a different reference dialect. Calcite will use a recording, if present.)
> * In each test, for each target dialect that has an *enabled* connection, execute the generated SQL against that connection. If there is a reference result for the test, check that the result equals the reference result.
> * In each test, for each target dialect that has a *recording* connection, execute the generated SQL against that connection. Record the outcome (error, column names and types, result) in the file. Proceed as for an enabled connection. (Check result against reference result, etc.)
> * In each test, for each target dialect that has a *replaying* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.)
> * In each test, for each target dialect that has a *replaying+recording* connection, look up the outcome of the generated SQL. If the outcome is present, proceed as for an enabled connection. (Check result against reference result, etc.) If the outcome is not present, proceed as for a recording connection. (Execute the SQL, record the outcome, check against reference result, etc.)
> With these improvements, the test suite:
> * Remains fast and robust (because recordings are used most of the time);
> * Can be run offline (using recordings);
> * Can be run by contributors who do not necessarily have access to all of the supported databases;
> * Has full coverage (runs all tests against all supported dialects);
> * Can re-generate the 'golden' references. (Must be online, have the required driver and connection, and takes more time.)
> Design:
> * A new {{class DialectTestConfig}} describes the dialects to be tested, their names, connections, the connection state (enabled, recording, replaying, replaying+recording), and recording file name.
> * A mock JDBC driver mediates recording/replay.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)