[DISCUSS] Running Sql Logic Tests for Calcite

[Mihai Budiu <mb...@gmail.com>]

Hello all,

I have submitted a PR for Calcite with a standalone executable that runs
the Sql Logic Test suite of 7+ million tests from sqlite.

This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
And this is the PR: https://github.com/apache/calcite/pull/3145

As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
general framework in Java to run these tests on any JDBC compliant
executor. So a question is whether this belongs to the Calcite project, or
some place else. sqlite is a C project, I didn't see any Java in their
source tree.

Please note that SQLite is in the public domain, so their licensing terms
are not an obstacle to using the test scripts.

The submitted code runs Calcite in its default configuration, but the
intent is for other projects that build Calcite-based compilers to be able
to test them by subclassing the "TestExecutors". In our own project (
https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
and we are not using the JDBC API.

The testsuite does find bugs in Calcite, both crashes and incorrect
results. So I think it's usefulness is not debated.

The second question is about the packaging of this program; right now it
has a main() entry point and it prints the results to stderr for human
consumption and triage. It is not clear to me how it should be inserted in
a CI infrastructure, since running all 7 million tests could take a long
time. One possible extension would be to have the program generate a
regression test for Calcite for each bug it finds, but I haven't
implemented this feature yet (and many failures could be due to the same
bug). But even that mode would not naturally integrate in a CI
infrastructure.

A simple possibility is for me to just publish the code as an independent
project on github with an MIT license (the code is derived from our
MIT-licensed project) and just advertise it to the Calcite community.

I would very much appreciate guidance.

Mihai Budiu

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[LakeShen <sh...@gmail.com>]

Thank you very much Mihai Budiu and Julian, we may use this test tool in
the future

Julian Hyde <jh...@apache.org> 于2023年4月30日周日 09:12写道：

> An update on this, for those of you not following
> https://issues.apache.org/jira/browse/CALCITE-5615. We agreed to move
> most of Mihai's code (except for the Calcite-specific code) into a new
> project, sql-logic-test. Today I made release 0.1 of that project, and
> published the artifacts to Maven Central.
>
> Thanks to Mihai and Stamatis for their contributions. The announcement is
> on Twitter: https://twitter.com/julianhyde/status/1652409133180817408
>
> The next step will be to rework 5615 to use the
> net.hydromatic:sql-logic-test library in Calcite's test suite.
>
> Julian
>
>
>
> On 2023/04/17 17:34:41 Julian Hyde wrote:
> > I agree with Stamatis that this has a similar “shape” to Quidem. I’d be
> happy to host the project under github.com/hydromatic. (If the maven
> group is net.hydromatic I can publish artifacts to Maven Central and
> Calcite could depend on those artifacts.)
> >
> > Regarding the frequency of testing. If we add it to CI and (say) 5% of
> the tests fail, I would find that demoralizing, even though passing 95% of
> the tests is actually a great achievement. So I would only deploy it as
> part of CI if there is a way to exclude failing tests.
> >
> > If the SqlLogicTest tool were defined in another repo, then there could
> be a Calcite module under plus [1] similar to TpchTest.
> >
> > Julian
> >
> > [1] https://github.com/apache/calcite/tree/main/plus
> >
> >
> >
> > > On Apr 17, 2023, at 1:58 AM, Stamatis Zampetakis <za...@gmail.com>
> wrote:
> > >
> > > Hey Mihai,
> > >
> > > Thanks for starting this discussion!
> > >
> > > Let's focus on the first question for now:
> > >
> > > Q1: Should the new slt module under PR-3145 [1] become part of Calcite
> > > repo or get its own?
> > >
> > > For those who have not followed the discussion under the CALCITE-5615
> > > [2] let me try to summarize a few things as per my understanding;
> > > Mihai can amend/correct things if necessary.
> > >
> > > The new slt module resembles a port of sqllogictest utility [3] to
> > > Java. It can parse and understand the test-script format used in
> > > sqllogictest and can run this scripts over JDBC compliant databases.
> > > It also accounts for extensions for Java engines without a JDBC
> > > interface.
> > >
> > > From my perspective, the code in [1] could perfectly stand on its own
> > > in a separate repo; there are already ports of sqllogictest in other
> > > languages such as Rust [4] and the latter appears to be quite popular.
> > > The sqllocitest parser/runner presents some similarities with the
> > > Quidem [5] executor that we are using for certain tests in Calcite.
> > > The Quidem project has its own repo although we are making use of it
> > > in Calcite.
> > > If it becomes a separate repo then the test scripts could also become
> > > part of the project making it more self-contained.
> > >
> > > On the other hand, we already have a testkit module in Calcite so
> > > bringing in new modules for testing purposes is relevant so why not
> > > slt as well. If it becomes part of Calcite it can get more visibility
> > > and facilitate maintenance since more people would be able to review
> > > and merge changes (not only Mihai).
> > >
> > > Since we are talking about a new module I would like to see some more
> > > people share their opinion on the topic before I continue the review.
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] https://github.com/apache/calcite/pull/3145
> > > [2] https://issues.apache.org/jira/browse/CALCITE-5615
> > > [3] https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
> > > [4] https://github.com/risinglightdb/sqllogictest-rs
> > > [5] https://github.com/julianhyde/quidem
> > >
> > >
> > >
> > > On Sat, Apr 15, 2023 at 11:31 AM Michael Mior <mm...@apache.org>
> wrote:
> > >>
> > >> Very cool! One approach could be to add set these tests to run
> periodically
> > >> (daily/weekly) as opposed to being part of the CI pipeline. That way
> we
> > >> still have a mechanism to keep tabs on bugs but the whole build isn't
> > >> slow/broken until this is fixed.
> > >>
> > >> On Fri, Apr 14, 2023, 15:20 Mihai Budiu <mb...@gmail.com> wrote:
> > >>
> > >>> Hello all,
> > >>>
> > >>> I have submitted a PR for Calcite with a standalone executable that
> runs
> > >>> the Sql Logic Test suite of 7+ million tests from sqlite.
> > >>>
> > >>> This is the JIRA case:
> https://issues.apache.org/jira/browse/CALCITE-5615
> > >>> And this is the PR: https://github.com/apache/calcite/pull/3145
> > >>>
> > >>> As Stamatis pointed out, the PR isn't really specific to Calcite, it
> is a
> > >>> general framework in Java to run these tests on any JDBC compliant
> > >>> executor. So a question is whether this belongs to the Calcite
> project, or
> > >>> some place else. sqlite is a C project, I didn't see any Java in
> their
> > >>> source tree.
> > >>>
> > >>> Please note that SQLite is in the public domain, so their licensing
> terms
> > >>> are not an obstacle to using the test scripts.
> > >>>
> > >>> The submitted code runs Calcite in its default configuration, but the
> > >>> intent is for other projects that build Calcite-based compilers to
> be able
> > >>> to test them by subclassing the "TestExecutors". In our own project (
> > >>> https://github.com/vmware/sql-to-dbsp-compiler) we have done
> exactly that,
> > >>> and we are not using the JDBC API.
> > >>>
> > >>> The testsuite does find bugs in Calcite, both crashes and incorrect
> > >>> results. So I think it's usefulness is not debated.
> > >>>
> > >>> The second question is about the packaging of this program; right
> now it
> > >>> has a main() entry point and it prints the results to stderr for
> human
> > >>> consumption and triage. It is not clear to me how it should be
> inserted in
> > >>> a CI infrastructure, since running all 7 million tests could take a
> long
> > >>> time. One possible extension would be to have the program generate a
> > >>> regression test for Calcite for each bug it finds, but I haven't
> > >>> implemented this feature yet (and many failures could be due to the
> same
> > >>> bug). But even that mode would not naturally integrate in a CI
> > >>> infrastructure.
> > >>>
> > >>> A simple possibility is for me to just publish the code as an
> independent
> > >>> project on github with an MIT license (the code is derived from our
> > >>> MIT-licensed project) and just advertise it to the Calcite community.
> > >>>
> > >>> I would very much appreciate guidance.
> > >>>
> > >>> Mihai Budiu
> > >>>
> >
> >
>

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[Julian Hyde <jh...@apache.org>]

An update on this, for those of you not following https://issues.apache.org/jira/browse/CALCITE-5615. We agreed to move most of Mihai's code (except for the Calcite-specific code) into a new project, sql-logic-test. Today I made release 0.1 of that project, and published the artifacts to Maven Central.

Thanks to Mihai and Stamatis for their contributions. The announcement is on Twitter: https://twitter.com/julianhyde/status/1652409133180817408

The next step will be to rework 5615 to use the net.hydromatic:sql-logic-test library in Calcite's test suite.

Julian



On 2023/04/17 17:34:41 Julian Hyde wrote:
> I agree with Stamatis that this has a similar “shape” to Quidem. I’d be happy to host the project under github.com/hydromatic. (If the maven group is net.hydromatic I can publish artifacts to Maven Central and Calcite could depend on those artifacts.)
> 
> Regarding the frequency of testing. If we add it to CI and (say) 5% of the tests fail, I would find that demoralizing, even though passing 95% of the tests is actually a great achievement. So I would only deploy it as part of CI if there is a way to exclude failing tests.
> 
> If the SqlLogicTest tool were defined in another repo, then there could be a Calcite module under plus [1] similar to TpchTest.
> 
> Julian
> 
> [1] https://github.com/apache/calcite/tree/main/plus 
> 
> 
> 
> > On Apr 17, 2023, at 1:58 AM, Stamatis Zampetakis <za...@gmail.com> wrote:
> > 
> > Hey Mihai,
> > 
> > Thanks for starting this discussion!
> > 
> > Let's focus on the first question for now:
> > 
> > Q1: Should the new slt module under PR-3145 [1] become part of Calcite
> > repo or get its own?
> > 
> > For those who have not followed the discussion under the CALCITE-5615
> > [2] let me try to summarize a few things as per my understanding;
> > Mihai can amend/correct things if necessary.
> > 
> > The new slt module resembles a port of sqllogictest utility [3] to
> > Java. It can parse and understand the test-script format used in
> > sqllogictest and can run this scripts over JDBC compliant databases.
> > It also accounts for extensions for Java engines without a JDBC
> > interface.
> > 
> > From my perspective, the code in [1] could perfectly stand on its own
> > in a separate repo; there are already ports of sqllogictest in other
> > languages such as Rust [4] and the latter appears to be quite popular.
> > The sqllocitest parser/runner presents some similarities with the
> > Quidem [5] executor that we are using for certain tests in Calcite.
> > The Quidem project has its own repo although we are making use of it
> > in Calcite.
> > If it becomes a separate repo then the test scripts could also become
> > part of the project making it more self-contained.
> > 
> > On the other hand, we already have a testkit module in Calcite so
> > bringing in new modules for testing purposes is relevant so why not
> > slt as well. If it becomes part of Calcite it can get more visibility
> > and facilitate maintenance since more people would be able to review
> > and merge changes (not only Mihai).
> > 
> > Since we are talking about a new module I would like to see some more
> > people share their opinion on the topic before I continue the review.
> > 
> > Best,
> > Stamatis
> > 
> > [1] https://github.com/apache/calcite/pull/3145
> > [2] https://issues.apache.org/jira/browse/CALCITE-5615
> > [3] https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
> > [4] https://github.com/risinglightdb/sqllogictest-rs
> > [5] https://github.com/julianhyde/quidem
> > 
> > 
> > 
> > On Sat, Apr 15, 2023 at 11:31 AM Michael Mior <mm...@apache.org> wrote:
> >> 
> >> Very cool! One approach could be to add set these tests to run periodically
> >> (daily/weekly) as opposed to being part of the CI pipeline. That way we
> >> still have a mechanism to keep tabs on bugs but the whole build isn't
> >> slow/broken until this is fixed.
> >> 
> >> On Fri, Apr 14, 2023, 15:20 Mihai Budiu <mb...@gmail.com> wrote:
> >> 
> >>> Hello all,
> >>> 
> >>> I have submitted a PR for Calcite with a standalone executable that runs
> >>> the Sql Logic Test suite of 7+ million tests from sqlite.
> >>> 
> >>> This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
> >>> And this is the PR: https://github.com/apache/calcite/pull/3145
> >>> 
> >>> As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
> >>> general framework in Java to run these tests on any JDBC compliant
> >>> executor. So a question is whether this belongs to the Calcite project, or
> >>> some place else. sqlite is a C project, I didn't see any Java in their
> >>> source tree.
> >>> 
> >>> Please note that SQLite is in the public domain, so their licensing terms
> >>> are not an obstacle to using the test scripts.
> >>> 
> >>> The submitted code runs Calcite in its default configuration, but the
> >>> intent is for other projects that build Calcite-based compilers to be able
> >>> to test them by subclassing the "TestExecutors". In our own project (
> >>> https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
> >>> and we are not using the JDBC API.
> >>> 
> >>> The testsuite does find bugs in Calcite, both crashes and incorrect
> >>> results. So I think it's usefulness is not debated.
> >>> 
> >>> The second question is about the packaging of this program; right now it
> >>> has a main() entry point and it prints the results to stderr for human
> >>> consumption and triage. It is not clear to me how it should be inserted in
> >>> a CI infrastructure, since running all 7 million tests could take a long
> >>> time. One possible extension would be to have the program generate a
> >>> regression test for Calcite for each bug it finds, but I haven't
> >>> implemented this feature yet (and many failures could be due to the same
> >>> bug). But even that mode would not naturally integrate in a CI
> >>> infrastructure.
> >>> 
> >>> A simple possibility is for me to just publish the code as an independent
> >>> project on github with an MIT license (the code is derived from our
> >>> MIT-licensed project) and just advertise it to the Calcite community.
> >>> 
> >>> I would very much appreciate guidance.
> >>> 
> >>> Mihai Budiu
> >>> 
> 
> 

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[Julian Hyde <jh...@gmail.com>]

I agree with Stamatis that this has a similar “shape” to Quidem. I’d be happy to host the project under github.com/hydromatic. (If the maven group is net.hydromatic I can publish artifacts to Maven Central and Calcite could depend on those artifacts.)

Regarding the frequency of testing. If we add it to CI and (say) 5% of the tests fail, I would find that demoralizing, even though passing 95% of the tests is actually a great achievement. So I would only deploy it as part of CI if there is a way to exclude failing tests.

If the SqlLogicTest tool were defined in another repo, then there could be a Calcite module under plus [1] similar to TpchTest.

Julian

[1] https://github.com/apache/calcite/tree/main/plus 



> On Apr 17, 2023, at 1:58 AM, Stamatis Zampetakis <za...@gmail.com> wrote:
> 
> Hey Mihai,
> 
> Thanks for starting this discussion!
> 
> Let's focus on the first question for now:
> 
> Q1: Should the new slt module under PR-3145 [1] become part of Calcite
> repo or get its own?
> 
> For those who have not followed the discussion under the CALCITE-5615
> [2] let me try to summarize a few things as per my understanding;
> Mihai can amend/correct things if necessary.
> 
> The new slt module resembles a port of sqllogictest utility [3] to
> Java. It can parse and understand the test-script format used in
> sqllogictest and can run this scripts over JDBC compliant databases.
> It also accounts for extensions for Java engines without a JDBC
> interface.
> 
> From my perspective, the code in [1] could perfectly stand on its own
> in a separate repo; there are already ports of sqllogictest in other
> languages such as Rust [4] and the latter appears to be quite popular.
> The sqllocitest parser/runner presents some similarities with the
> Quidem [5] executor that we are using for certain tests in Calcite.
> The Quidem project has its own repo although we are making use of it
> in Calcite.
> If it becomes a separate repo then the test scripts could also become
> part of the project making it more self-contained.
> 
> On the other hand, we already have a testkit module in Calcite so
> bringing in new modules for testing purposes is relevant so why not
> slt as well. If it becomes part of Calcite it can get more visibility
> and facilitate maintenance since more people would be able to review
> and merge changes (not only Mihai).
> 
> Since we are talking about a new module I would like to see some more
> people share their opinion on the topic before I continue the review.
> 
> Best,
> Stamatis
> 
> [1] https://github.com/apache/calcite/pull/3145
> [2] https://issues.apache.org/jira/browse/CALCITE-5615
> [3] https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
> [4] https://github.com/risinglightdb/sqllogictest-rs
> [5] https://github.com/julianhyde/quidem
> 
> 
> 
> On Sat, Apr 15, 2023 at 11:31 AM Michael Mior <mm...@apache.org> wrote:
>> 
>> Very cool! One approach could be to add set these tests to run periodically
>> (daily/weekly) as opposed to being part of the CI pipeline. That way we
>> still have a mechanism to keep tabs on bugs but the whole build isn't
>> slow/broken until this is fixed.
>> 
>> On Fri, Apr 14, 2023, 15:20 Mihai Budiu <mb...@gmail.com> wrote:
>> 
>>> Hello all,
>>> 
>>> I have submitted a PR for Calcite with a standalone executable that runs
>>> the Sql Logic Test suite of 7+ million tests from sqlite.
>>> 
>>> This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
>>> And this is the PR: https://github.com/apache/calcite/pull/3145
>>> 
>>> As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
>>> general framework in Java to run these tests on any JDBC compliant
>>> executor. So a question is whether this belongs to the Calcite project, or
>>> some place else. sqlite is a C project, I didn't see any Java in their
>>> source tree.
>>> 
>>> Please note that SQLite is in the public domain, so their licensing terms
>>> are not an obstacle to using the test scripts.
>>> 
>>> The submitted code runs Calcite in its default configuration, but the
>>> intent is for other projects that build Calcite-based compilers to be able
>>> to test them by subclassing the "TestExecutors". In our own project (
>>> https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
>>> and we are not using the JDBC API.
>>> 
>>> The testsuite does find bugs in Calcite, both crashes and incorrect
>>> results. So I think it's usefulness is not debated.
>>> 
>>> The second question is about the packaging of this program; right now it
>>> has a main() entry point and it prints the results to stderr for human
>>> consumption and triage. It is not clear to me how it should be inserted in
>>> a CI infrastructure, since running all 7 million tests could take a long
>>> time. One possible extension would be to have the program generate a
>>> regression test for Calcite for each bug it finds, but I haven't
>>> implemented this feature yet (and many failures could be due to the same
>>> bug). But even that mode would not naturally integrate in a CI
>>> infrastructure.
>>> 
>>> A simple possibility is for me to just publish the code as an independent
>>> project on github with an MIT license (the code is derived from our
>>> MIT-licensed project) and just advertise it to the Calcite community.
>>> 
>>> I would very much appreciate guidance.
>>> 
>>> Mihai Budiu
>>> 

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[Stamatis Zampetakis <za...@gmail.com>]

Hey Mihai,

Thanks for starting this discussion!

Let's focus on the first question for now:

Q1: Should the new slt module under PR-3145 [1] become part of Calcite
repo or get its own?

For those who have not followed the discussion under the CALCITE-5615
[2] let me try to summarize a few things as per my understanding;
Mihai can amend/correct things if necessary.

The new slt module resembles a port of sqllogictest utility [3] to
Java. It can parse and understand the test-script format used in
sqllogictest and can run this scripts over JDBC compliant databases.
It also accounts for extensions for Java engines without a JDBC
interface.

From my perspective, the code in [1] could perfectly stand on its own
in a separate repo; there are already ports of sqllogictest in other
languages such as Rust [4] and the latter appears to be quite popular.
The sqllocitest parser/runner presents some similarities with the
Quidem [5] executor that we are using for certain tests in Calcite.
The Quidem project has its own repo although we are making use of it
in Calcite.
If it becomes a separate repo then the test scripts could also become
part of the project making it more self-contained.

On the other hand, we already have a testkit module in Calcite so
bringing in new modules for testing purposes is relevant so why not
slt as well. If it becomes part of Calcite it can get more visibility
and facilitate maintenance since more people would be able to review
and merge changes (not only Mihai).

Since we are talking about a new module I would like to see some more
people share their opinion on the topic before I continue the review.

Best,
Stamatis

[1] https://github.com/apache/calcite/pull/3145
[2] https://issues.apache.org/jira/browse/CALCITE-5615
[3] https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
[4] https://github.com/risinglightdb/sqllogictest-rs
[5] https://github.com/julianhyde/quidem



On Sat, Apr 15, 2023 at 11:31 AM Michael Mior <mm...@apache.org> wrote:
>
> Very cool! One approach could be to add set these tests to run periodically
> (daily/weekly) as opposed to being part of the CI pipeline. That way we
> still have a mechanism to keep tabs on bugs but the whole build isn't
> slow/broken until this is fixed.
>
> On Fri, Apr 14, 2023, 15:20 Mihai Budiu <mb...@gmail.com> wrote:
>
> > Hello all,
> >
> > I have submitted a PR for Calcite with a standalone executable that runs
> > the Sql Logic Test suite of 7+ million tests from sqlite.
> >
> > This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
> > And this is the PR: https://github.com/apache/calcite/pull/3145
> >
> > As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
> > general framework in Java to run these tests on any JDBC compliant
> > executor. So a question is whether this belongs to the Calcite project, or
> > some place else. sqlite is a C project, I didn't see any Java in their
> > source tree.
> >
> > Please note that SQLite is in the public domain, so their licensing terms
> > are not an obstacle to using the test scripts.
> >
> > The submitted code runs Calcite in its default configuration, but the
> > intent is for other projects that build Calcite-based compilers to be able
> > to test them by subclassing the "TestExecutors". In our own project (
> > https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
> > and we are not using the JDBC API.
> >
> > The testsuite does find bugs in Calcite, both crashes and incorrect
> > results. So I think it's usefulness is not debated.
> >
> > The second question is about the packaging of this program; right now it
> > has a main() entry point and it prints the results to stderr for human
> > consumption and triage. It is not clear to me how it should be inserted in
> > a CI infrastructure, since running all 7 million tests could take a long
> > time. One possible extension would be to have the program generate a
> > regression test for Calcite for each bug it finds, but I haven't
> > implemented this feature yet (and many failures could be due to the same
> > bug). But even that mode would not naturally integrate in a CI
> > infrastructure.
> >
> > A simple possibility is for me to just publish the code as an independent
> > project on github with an MIT license (the code is derived from our
> > MIT-licensed project) and just advertise it to the Calcite community.
> >
> > I would very much appreciate guidance.
> >
> > Mihai Budiu
> >

Re: [DISCUSS] Running Sql Logic Tests for Calcite

[Michael Mior <mm...@apache.org>]

Very cool! One approach could be to add set these tests to run periodically
(daily/weekly) as opposed to being part of the CI pipeline. That way we
still have a mechanism to keep tabs on bugs but the whole build isn't
slow/broken until this is fixed.

On Fri, Apr 14, 2023, 15:20 Mihai Budiu <mb...@gmail.com> wrote:

> Hello all,
>
> I have submitted a PR for Calcite with a standalone executable that runs
> the Sql Logic Test suite of 7+ million tests from sqlite.
>
> This is the JIRA case: https://issues.apache.org/jira/browse/CALCITE-5615
> And this is the PR: https://github.com/apache/calcite/pull/3145
>
> As Stamatis pointed out, the PR isn't really specific to Calcite, it is a
> general framework in Java to run these tests on any JDBC compliant
> executor. So a question is whether this belongs to the Calcite project, or
> some place else. sqlite is a C project, I didn't see any Java in their
> source tree.
>
> Please note that SQLite is in the public domain, so their licensing terms
> are not an obstacle to using the test scripts.
>
> The submitted code runs Calcite in its default configuration, but the
> intent is for other projects that build Calcite-based compilers to be able
> to test them by subclassing the "TestExecutors". In our own project (
> https://github.com/vmware/sql-to-dbsp-compiler) we have done exactly that,
> and we are not using the JDBC API.
>
> The testsuite does find bugs in Calcite, both crashes and incorrect
> results. So I think it's usefulness is not debated.
>
> The second question is about the packaging of this program; right now it
> has a main() entry point and it prints the results to stderr for human
> consumption and triage. It is not clear to me how it should be inserted in
> a CI infrastructure, since running all 7 million tests could take a long
> time. One possible extension would be to have the program generate a
> regression test for Calcite for each bug it finds, but I haven't
> implemented this feature yet (and many failures could be due to the same
> bug). But even that mode would not naturally integrate in a CI
> infrastructure.
>
> A simple possibility is for me to just publish the code as an independent
> project on github with an MIT license (the code is derived from our
> MIT-licensed project) and just advertise it to the Calcite community.
>
> I would very much appreciate guidance.
>
> Mihai Budiu
>