You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dmitri Bourlatchkov <di...@apache.org> on 2026/05/28 21:07:44 UTC
[DISCUSS] Future of the "regtests" code
Hi All,
This is to recap and follow up on today's Community Sync discussion. This
matter came up during the review of the Spark 4 PR, but I believe it
deserves a dedicated discussion.
Observations:
* Tests under the /regtests directory take a considerable amount of CI
resources. This is mostly due to building custom Docker images.
* When adding support for newer Spark versions, developers naturally tend
to copy existing test infrastructure, which results in building more Docker
images.
* The coverage these tests provide probably does not require the heavy
docker machinery. These tests validate that the Spark Session can interact
with Polaris over the Iceberg REST Catalog Java client. The same coverage
can be provided in an isolated JVM running under Gradle more efficiently
(without a docker env.)
Proposal:
* Gradually convert Spark tests under /regtest to Gradle tasks and JUnit
tests without Docker.
The exact impl. is to be figured out.
Most of the tests can run under the Junit framework using a local Polaris
server (same JVM or different JVM depending on use case).
True "integration" tests can still run under Gradle as a task that starts a
Spark shell in a fresh JVM and executes a small set of SQL commands in it.
However, these "big" tests can probably be limited in number and complexity
and as such should not generate excessive CI load. More specifically, these
tests probably do not need to assert the verbatim output of the SQL
commands. A basic sanity check should be sufficient. Functional tests can
be performed under JUnit, I think.
Thoughts?
Thanks,
Dmitri.
Re: [DISCUSS] Future of the "regtests" code
Posted by Yufei Gu <fl...@gmail.com>.
Hi folks,
I'm all for keeping things simple. I agree that we should refactor the
regtests to some extent. For example, we've discussed enabling real AWS S3
tests for quite a while. Maybe JB and I can help restart that effort.
I looked at a recent CI run from PR #4535. Based on that example, moving
regtests to integration tests would not necessarily save time. In fact, it
could potentially increase overall CI duration, since the longest running
workflows are currently not the regtests.
CI/PR / CI/PR / Regression Tests (minio) (pull_request)Successful in 20m
CI/PR / CI/PR / Regression Tests (rustfs) (pull_request)Successful in 21m
CI/PR / CI/PR / Runtime Service Integration Tests (pull_request)Successful
in 24m
CI/PR / CI/PR / Other Integration Tests (pull_request)Successful in 25m
Given that, I think it would be helpful to first quantify the expected CI
savings before we invest significant effort in migrating the existing
regtests.
Yufei
On Mon, Jun 1, 2026 at 2:51 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
> Hi All,
>
> Reusing the standard Polaris server images produced during CI for
> integration tests sound reasonable. I hope these tests will not cause too
> much extra CI overhead.
>
> However, testing various _clients_ is probably not worth the extra CI time.
> Polaris presents a REST API to clients. If a particular client is known to
> expect a particular behaviour from the server, such an expectation can be
> coded in java, under the JUnit framework.
>
> If we do not know what specific behaviours current regtests are expected to
> validate, that alone is probably not a strong argument for keeping them. If
> we cannot accept the risk of altering existing regtests, they become a
> maintenance burden.
>
> How about moving the existing regtests to a nightly/weekly schedule based
> on `main` and gradually migrating their "logic" to Gradle/CI tests? If
> regressions are detected, we'll enhance those new tests to cover the same
> expectations. WDYT?
>
> Thanks.
> Dmitri.
>
>
>
> On Mon, Jun 1, 2026 at 9:15 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi all,
> >
> > Currently, the regtests can consume 40+ minutes for PR CI runs, plus CI
> for
> > the main and release branches.
> > That's a significant cost, so I think we should have a good purpose for
> > running those every time CI is triggered.
> >
> > The Spark plugin regtests have, at least to what I can see, coverage that
> > looks narrow enough that it could mostly be implemented as JUnit/Gradle
> > integration tests.
> >
> > On the top-level regtests:
> > The cloud-provider specific tests do not appear to run, because we do not
> > have credentials to run those.
> > I could not find any tests that run against another query engine than
> Spark
> > or PySpark.
> >
> > I could imagine reducing Docker usage to a minimum, and, for example, use
> > the Polaris container image that is already build in CI.
> > This can also be used for the Python CLI, if there is no better
> > alternative.
> >
> > I am not saying that end-2-end tests are not useful.
> > But I would like to keep the scope of these regtests to be small,
> > purposeful, and isolated from ordinary CI cost, unless they protect a
> > distinct integration boundary.
> > Then we could add other query engines, which are not yet covered, as
> well.
> > If we want broader engine coverage, such as Trino or DuckDB, that should
> be
> > explicit and targeted rather than a reason to keep duplicating the
> current
> > Spark SQL regtest stack.
> >
> > Robert
> >
> >
> > On Sun, May 31, 2026 at 1:33 AM Michael Collado <co...@gmail.com>
> > wrote:
> >
> > > There are a number of times the e2e purest have caught issues the have
> > > tests don’t cover. Personally, I’d love to see expansion of the tests
> to
> > > cover frameworks that we simply can’t cover in the Java tests (pyspark,
> > > duckdb, trino, etc.). The variance in support across engines and
> catalogs
> > > was kind of a theme at the Iceberg Summit in April. Broader coverage of
> > > testing in the ecosystem would really help Polaris in that regard.
> > >
> > > Mike
> > >
> > > On Thu, May 28, 2026 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Yong,
> > > >
> > > > Good questions :) As for me I'd refactor all shell-based regtests
> that
> > > use
> > > > Docker into Gradle-based tests (preferably JUnit).
> > > >
> > > > If people find value in the section of the regtests that talk to real
> > > > cloud services (but do not run in CI), we can keep them. Still, we
> have
> > > > examples of JUnit-based tests that can talk to real cloud services
> too.
> > > >
> > > > However, this is just my opinion. Let's wait a bit for other people
> to
> > > > express their opinions. We had a good discussion during the sync
> call,
> > > but
> > > > I'm not sure we have consensus yet... hence this thread.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org>
> wrote:
> > > >
> > > > > One clarification, the regtests we talk about here is only
> > > > >
> > >
> https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
> > > > > or https://github.com/apache/polaris/tree/main/regtests as well?
> as
> > > the
> > > > > v3.5/regtests doesn't have cloud dependencies but the later one
> has.
> > > > >
> > > > > Thanks,
> > > > > Yong Zheng
> > > > >
> > > > > On 2026/05/29 01:29:24 Yong Zheng wrote:
> > > > > > Hello Dmitri,
> > > > > >
> > > > > > Thanks for the recap. If I understand correctly, we want to merge
> > > test
> > > > > coverage from `/regtests` (docker based) into `/integration`?
> > Looking
> > > at
> > > > > them, integration has most of the tests we have in regtests excepts
> > the
> > > > > ones that are real cloud specific such as spark_sql_gcp* and
> > > > > spark_sql_azure* as well as pyspark specific (t_pyspark). The
> cloud
> > > > > specific ones are not part of CI, in that case, do we still want to
> > > > covert
> > > > > them? Also, I am not sure if we can convert the pyspark code into
> > JUnit
> > > > > directly.
> > > > > >
> > > > > > Also, regarding those duplicates classes, should we move them to
> > > > common?
> > > > > For the ones that are very similar but minor diff, should we
> proceed
> > > with
> > > > > adapters for version specific (meaning, we won't be follow what
> > Iceberg
> > > > is
> > > > > doing with different version of spark).
> > > > > >
> > > > > > Lastly, should we close current PR and handle the two above in a
> > > > > separate PRs first before revisiting Spark4 support?
> > > > > >
> > > > > > Thanks,
> > > > > > Yong Zheng
> > > > > >
> > > > > > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > > > > > > Hi All,
> > > > > > >
> > > > > > > This is to recap and follow up on today's Community Sync
> > > discussion.
> > > > > This
> > > > > > > matter came up during the review of the Spark 4 PR, but I
> believe
> > > it
> > > > > > > deserves a dedicated discussion.
> > > > > > >
> > > > > > > Observations:
> > > > > > >
> > > > > > > * Tests under the /regtests directory take a considerable
> amount
> > of
> > > > CI
> > > > > > > resources. This is mostly due to building custom Docker images.
> > > > > > >
> > > > > > > * When adding support for newer Spark versions, developers
> > > naturally
> > > > > tend
> > > > > > > to copy existing test infrastructure, which results in building
> > > more
> > > > > Docker
> > > > > > > images.
> > > > > > >
> > > > > > > * The coverage these tests provide probably does not require
> the
> > > > heavy
> > > > > > > docker machinery. These tests validate that the Spark Session
> can
> > > > > interact
> > > > > > > with Polaris over the Iceberg REST Catalog Java client. The
> same
> > > > > coverage
> > > > > > > can be provided in an isolated JVM running under Gradle more
> > > > > efficiently
> > > > > > > (without a docker env.)
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > > * Gradually convert Spark tests under /regtest to Gradle tasks
> > and
> > > > > JUnit
> > > > > > > tests without Docker.
> > > > > > >
> > > > > > > The exact impl. is to be figured out.
> > > > > > >
> > > > > > > Most of the tests can run under the Junit framework using a
> local
> > > > > Polaris
> > > > > > > server (same JVM or different JVM depending on use case).
> > > > > > >
> > > > > > > True "integration" tests can still run under Gradle as a task
> > that
> > > > > starts a
> > > > > > > Spark shell in a fresh JVM and executes a small set of SQL
> > commands
> > > > in
> > > > > it.
> > > > > > > However, these "big" tests can probably be limited in number
> and
> > > > > complexity
> > > > > > > and as such should not generate excessive CI load. More
> > > specifically,
> > > > > these
> > > > > > > tests probably do not need to assert the verbatim output of the
> > SQL
> > > > > > > commands. A basic sanity check should be sufficient. Functional
> > > tests
> > > > > can
> > > > > > > be performed under JUnit, I think.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
Reusing the standard Polaris server images produced during CI for
integration tests sound reasonable. I hope these tests will not cause too
much extra CI overhead.
However, testing various _clients_ is probably not worth the extra CI time.
Polaris presents a REST API to clients. If a particular client is known to
expect a particular behaviour from the server, such an expectation can be
coded in java, under the JUnit framework.
If we do not know what specific behaviours current regtests are expected to
validate, that alone is probably not a strong argument for keeping them. If
we cannot accept the risk of altering existing regtests, they become a
maintenance burden.
How about moving the existing regtests to a nightly/weekly schedule based
on `main` and gradually migrating their "logic" to Gradle/CI tests? If
regressions are detected, we'll enhance those new tests to cover the same
expectations. WDYT?
Thanks.
Dmitri.
On Mon, Jun 1, 2026 at 9:15 AM Robert Stupp <sn...@snazy.de> wrote:
> Hi all,
>
> Currently, the regtests can consume 40+ minutes for PR CI runs, plus CI for
> the main and release branches.
> That's a significant cost, so I think we should have a good purpose for
> running those every time CI is triggered.
>
> The Spark plugin regtests have, at least to what I can see, coverage that
> looks narrow enough that it could mostly be implemented as JUnit/Gradle
> integration tests.
>
> On the top-level regtests:
> The cloud-provider specific tests do not appear to run, because we do not
> have credentials to run those.
> I could not find any tests that run against another query engine than Spark
> or PySpark.
>
> I could imagine reducing Docker usage to a minimum, and, for example, use
> the Polaris container image that is already build in CI.
> This can also be used for the Python CLI, if there is no better
> alternative.
>
> I am not saying that end-2-end tests are not useful.
> But I would like to keep the scope of these regtests to be small,
> purposeful, and isolated from ordinary CI cost, unless they protect a
> distinct integration boundary.
> Then we could add other query engines, which are not yet covered, as well.
> If we want broader engine coverage, such as Trino or DuckDB, that should be
> explicit and targeted rather than a reason to keep duplicating the current
> Spark SQL regtest stack.
>
> Robert
>
>
> On Sun, May 31, 2026 at 1:33 AM Michael Collado <co...@gmail.com>
> wrote:
>
> > There are a number of times the e2e purest have caught issues the have
> > tests don’t cover. Personally, I’d love to see expansion of the tests to
> > cover frameworks that we simply can’t cover in the Java tests (pyspark,
> > duckdb, trino, etc.). The variance in support across engines and catalogs
> > was kind of a theme at the Iceberg Summit in April. Broader coverage of
> > testing in the ecosystem would really help Polaris in that regard.
> >
> > Mike
> >
> > On Thu, May 28, 2026 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Yong,
> > >
> > > Good questions :) As for me I'd refactor all shell-based regtests that
> > use
> > > Docker into Gradle-based tests (preferably JUnit).
> > >
> > > If people find value in the section of the regtests that talk to real
> > > cloud services (but do not run in CI), we can keep them. Still, we have
> > > examples of JUnit-based tests that can talk to real cloud services too.
> > >
> > > However, this is just my opinion. Let's wait a bit for other people to
> > > express their opinions. We had a good discussion during the sync call,
> > but
> > > I'm not sure we have consensus yet... hence this thread.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org> wrote:
> > >
> > > > One clarification, the regtests we talk about here is only
> > > >
> > https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
> > > > or https://github.com/apache/polaris/tree/main/regtests as well? as
> > the
> > > > v3.5/regtests doesn't have cloud dependencies but the later one has.
> > > >
> > > > Thanks,
> > > > Yong Zheng
> > > >
> > > > On 2026/05/29 01:29:24 Yong Zheng wrote:
> > > > > Hello Dmitri,
> > > > >
> > > > > Thanks for the recap. If I understand correctly, we want to merge
> > test
> > > > coverage from `/regtests` (docker based) into `/integration`?
> Looking
> > at
> > > > them, integration has most of the tests we have in regtests excepts
> the
> > > > ones that are real cloud specific such as spark_sql_gcp* and
> > > > spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud
> > > > specific ones are not part of CI, in that case, do we still want to
> > > covert
> > > > them? Also, I am not sure if we can convert the pyspark code into
> JUnit
> > > > directly.
> > > > >
> > > > > Also, regarding those duplicates classes, should we move them to
> > > common?
> > > > For the ones that are very similar but minor diff, should we proceed
> > with
> > > > adapters for version specific (meaning, we won't be follow what
> Iceberg
> > > is
> > > > doing with different version of spark).
> > > > >
> > > > > Lastly, should we close current PR and handle the two above in a
> > > > separate PRs first before revisiting Spark4 support?
> > > > >
> > > > > Thanks,
> > > > > Yong Zheng
> > > > >
> > > > > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > > > > > Hi All,
> > > > > >
> > > > > > This is to recap and follow up on today's Community Sync
> > discussion.
> > > > This
> > > > > > matter came up during the review of the Spark 4 PR, but I believe
> > it
> > > > > > deserves a dedicated discussion.
> > > > > >
> > > > > > Observations:
> > > > > >
> > > > > > * Tests under the /regtests directory take a considerable amount
> of
> > > CI
> > > > > > resources. This is mostly due to building custom Docker images.
> > > > > >
> > > > > > * When adding support for newer Spark versions, developers
> > naturally
> > > > tend
> > > > > > to copy existing test infrastructure, which results in building
> > more
> > > > Docker
> > > > > > images.
> > > > > >
> > > > > > * The coverage these tests provide probably does not require the
> > > heavy
> > > > > > docker machinery. These tests validate that the Spark Session can
> > > > interact
> > > > > > with Polaris over the Iceberg REST Catalog Java client. The same
> > > > coverage
> > > > > > can be provided in an isolated JVM running under Gradle more
> > > > efficiently
> > > > > > (without a docker env.)
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > > * Gradually convert Spark tests under /regtest to Gradle tasks
> and
> > > > JUnit
> > > > > > tests without Docker.
> > > > > >
> > > > > > The exact impl. is to be figured out.
> > > > > >
> > > > > > Most of the tests can run under the Junit framework using a local
> > > > Polaris
> > > > > > server (same JVM or different JVM depending on use case).
> > > > > >
> > > > > > True "integration" tests can still run under Gradle as a task
> that
> > > > starts a
> > > > > > Spark shell in a fresh JVM and executes a small set of SQL
> commands
> > > in
> > > > it.
> > > > > > However, these "big" tests can probably be limited in number and
> > > > complexity
> > > > > > and as such should not generate excessive CI load. More
> > specifically,
> > > > these
> > > > > > tests probably do not need to assert the verbatim output of the
> SQL
> > > > > > commands. A basic sanity check should be sufficient. Functional
> > tests
> > > > can
> > > > > > be performed under JUnit, I think.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Robert Stupp <sn...@snazy.de>.
Hi all,
Currently, the regtests can consume 40+ minutes for PR CI runs, plus CI for
the main and release branches.
That's a significant cost, so I think we should have a good purpose for
running those every time CI is triggered.
The Spark plugin regtests have, at least to what I can see, coverage that
looks narrow enough that it could mostly be implemented as JUnit/Gradle
integration tests.
On the top-level regtests:
The cloud-provider specific tests do not appear to run, because we do not
have credentials to run those.
I could not find any tests that run against another query engine than Spark
or PySpark.
I could imagine reducing Docker usage to a minimum, and, for example, use
the Polaris container image that is already build in CI.
This can also be used for the Python CLI, if there is no better alternative.
I am not saying that end-2-end tests are not useful.
But I would like to keep the scope of these regtests to be small,
purposeful, and isolated from ordinary CI cost, unless they protect a
distinct integration boundary.
Then we could add other query engines, which are not yet covered, as well.
If we want broader engine coverage, such as Trino or DuckDB, that should be
explicit and targeted rather than a reason to keep duplicating the current
Spark SQL regtest stack.
Robert
On Sun, May 31, 2026 at 1:33 AM Michael Collado <co...@gmail.com>
wrote:
> There are a number of times the e2e purest have caught issues the have
> tests don’t cover. Personally, I’d love to see expansion of the tests to
> cover frameworks that we simply can’t cover in the Java tests (pyspark,
> duckdb, trino, etc.). The variance in support across engines and catalogs
> was kind of a theme at the Iceberg Summit in April. Broader coverage of
> testing in the ecosystem would really help Polaris in that regard.
>
> Mike
>
> On Thu, May 28, 2026 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yong,
> >
> > Good questions :) As for me I'd refactor all shell-based regtests that
> use
> > Docker into Gradle-based tests (preferably JUnit).
> >
> > If people find value in the section of the regtests that talk to real
> > cloud services (but do not run in CI), we can keep them. Still, we have
> > examples of JUnit-based tests that can talk to real cloud services too.
> >
> > However, this is just my opinion. Let's wait a bit for other people to
> > express their opinions. We had a good discussion during the sync call,
> but
> > I'm not sure we have consensus yet... hence this thread.
> >
> > Cheers,
> > Dmitri.
> >
> > On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org> wrote:
> >
> > > One clarification, the regtests we talk about here is only
> > >
> https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
> > > or https://github.com/apache/polaris/tree/main/regtests as well? as
> the
> > > v3.5/regtests doesn't have cloud dependencies but the later one has.
> > >
> > > Thanks,
> > > Yong Zheng
> > >
> > > On 2026/05/29 01:29:24 Yong Zheng wrote:
> > > > Hello Dmitri,
> > > >
> > > > Thanks for the recap. If I understand correctly, we want to merge
> test
> > > coverage from `/regtests` (docker based) into `/integration`? Looking
> at
> > > them, integration has most of the tests we have in regtests excepts the
> > > ones that are real cloud specific such as spark_sql_gcp* and
> > > spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud
> > > specific ones are not part of CI, in that case, do we still want to
> > covert
> > > them? Also, I am not sure if we can convert the pyspark code into JUnit
> > > directly.
> > > >
> > > > Also, regarding those duplicates classes, should we move them to
> > common?
> > > For the ones that are very similar but minor diff, should we proceed
> with
> > > adapters for version specific (meaning, we won't be follow what Iceberg
> > is
> > > doing with different version of spark).
> > > >
> > > > Lastly, should we close current PR and handle the two above in a
> > > separate PRs first before revisiting Spark4 support?
> > > >
> > > > Thanks,
> > > > Yong Zheng
> > > >
> > > > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > > > > Hi All,
> > > > >
> > > > > This is to recap and follow up on today's Community Sync
> discussion.
> > > This
> > > > > matter came up during the review of the Spark 4 PR, but I believe
> it
> > > > > deserves a dedicated discussion.
> > > > >
> > > > > Observations:
> > > > >
> > > > > * Tests under the /regtests directory take a considerable amount of
> > CI
> > > > > resources. This is mostly due to building custom Docker images.
> > > > >
> > > > > * When adding support for newer Spark versions, developers
> naturally
> > > tend
> > > > > to copy existing test infrastructure, which results in building
> more
> > > Docker
> > > > > images.
> > > > >
> > > > > * The coverage these tests provide probably does not require the
> > heavy
> > > > > docker machinery. These tests validate that the Spark Session can
> > > interact
> > > > > with Polaris over the Iceberg REST Catalog Java client. The same
> > > coverage
> > > > > can be provided in an isolated JVM running under Gradle more
> > > efficiently
> > > > > (without a docker env.)
> > > > >
> > > > > Proposal:
> > > > >
> > > > > * Gradually convert Spark tests under /regtest to Gradle tasks and
> > > JUnit
> > > > > tests without Docker.
> > > > >
> > > > > The exact impl. is to be figured out.
> > > > >
> > > > > Most of the tests can run under the Junit framework using a local
> > > Polaris
> > > > > server (same JVM or different JVM depending on use case).
> > > > >
> > > > > True "integration" tests can still run under Gradle as a task that
> > > starts a
> > > > > Spark shell in a fresh JVM and executes a small set of SQL commands
> > in
> > > it.
> > > > > However, these "big" tests can probably be limited in number and
> > > complexity
> > > > > and as such should not generate excessive CI load. More
> specifically,
> > > these
> > > > > tests probably do not need to assert the verbatim output of the SQL
> > > > > commands. A basic sanity check should be sufficient. Functional
> tests
> > > can
> > > > > be performed under JUnit, I think.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Michael Collado <co...@gmail.com>.
There are a number of times the e2e purest have caught issues the have
tests don’t cover. Personally, I’d love to see expansion of the tests to
cover frameworks that we simply can’t cover in the Java tests (pyspark,
duckdb, trino, etc.). The variance in support across engines and catalogs
was kind of a theme at the Iceberg Summit in April. Broader coverage of
testing in the ecosystem would really help Polaris in that regard.
Mike
On Thu, May 28, 2026 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yong,
>
> Good questions :) As for me I'd refactor all shell-based regtests that use
> Docker into Gradle-based tests (preferably JUnit).
>
> If people find value in the section of the regtests that talk to real
> cloud services (but do not run in CI), we can keep them. Still, we have
> examples of JUnit-based tests that can talk to real cloud services too.
>
> However, this is just my opinion. Let's wait a bit for other people to
> express their opinions. We had a good discussion during the sync call, but
> I'm not sure we have consensus yet... hence this thread.
>
> Cheers,
> Dmitri.
>
> On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org> wrote:
>
> > One clarification, the regtests we talk about here is only
> > https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
> > or https://github.com/apache/polaris/tree/main/regtests as well? as the
> > v3.5/regtests doesn't have cloud dependencies but the later one has.
> >
> > Thanks,
> > Yong Zheng
> >
> > On 2026/05/29 01:29:24 Yong Zheng wrote:
> > > Hello Dmitri,
> > >
> > > Thanks for the recap. If I understand correctly, we want to merge test
> > coverage from `/regtests` (docker based) into `/integration`? Looking at
> > them, integration has most of the tests we have in regtests excepts the
> > ones that are real cloud specific such as spark_sql_gcp* and
> > spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud
> > specific ones are not part of CI, in that case, do we still want to
> covert
> > them? Also, I am not sure if we can convert the pyspark code into JUnit
> > directly.
> > >
> > > Also, regarding those duplicates classes, should we move them to
> common?
> > For the ones that are very similar but minor diff, should we proceed with
> > adapters for version specific (meaning, we won't be follow what Iceberg
> is
> > doing with different version of spark).
> > >
> > > Lastly, should we close current PR and handle the two above in a
> > separate PRs first before revisiting Spark4 support?
> > >
> > > Thanks,
> > > Yong Zheng
> > >
> > > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > > > Hi All,
> > > >
> > > > This is to recap and follow up on today's Community Sync discussion.
> > This
> > > > matter came up during the review of the Spark 4 PR, but I believe it
> > > > deserves a dedicated discussion.
> > > >
> > > > Observations:
> > > >
> > > > * Tests under the /regtests directory take a considerable amount of
> CI
> > > > resources. This is mostly due to building custom Docker images.
> > > >
> > > > * When adding support for newer Spark versions, developers naturally
> > tend
> > > > to copy existing test infrastructure, which results in building more
> > Docker
> > > > images.
> > > >
> > > > * The coverage these tests provide probably does not require the
> heavy
> > > > docker machinery. These tests validate that the Spark Session can
> > interact
> > > > with Polaris over the Iceberg REST Catalog Java client. The same
> > coverage
> > > > can be provided in an isolated JVM running under Gradle more
> > efficiently
> > > > (without a docker env.)
> > > >
> > > > Proposal:
> > > >
> > > > * Gradually convert Spark tests under /regtest to Gradle tasks and
> > JUnit
> > > > tests without Docker.
> > > >
> > > > The exact impl. is to be figured out.
> > > >
> > > > Most of the tests can run under the Junit framework using a local
> > Polaris
> > > > server (same JVM or different JVM depending on use case).
> > > >
> > > > True "integration" tests can still run under Gradle as a task that
> > starts a
> > > > Spark shell in a fresh JVM and executes a small set of SQL commands
> in
> > it.
> > > > However, these "big" tests can probably be limited in number and
> > complexity
> > > > and as such should not generate excessive CI load. More specifically,
> > these
> > > > tests probably do not need to assert the verbatim output of the SQL
> > > > commands. A basic sanity check should be sufficient. Functional tests
> > can
> > > > be performed under JUnit, I think.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > >
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,
Many thanks to Yong for working on PR [4588]!
This PR performs a Spark smoke test using both the "fat jar" and the "jar
with dependencies" artifacts of the Polaris Spark Client. IMHO, the
coverage this test offers is generally equivalent to the coverage we got
from the docker-base regtests. At the same time, the execution of this test
can be expected to be faster and debugging easier.
I approved [4588] in GH and invite other interested reviewers to take a
look too.
[4588] https://github.com/apache/polaris/pull/4588
Cheers,
Dmitri.
On Thu, May 28, 2026 at 10:58 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yong,
>
> Good questions :) As for me I'd refactor all shell-based regtests that use
> Docker into Gradle-based tests (preferably JUnit).
>
> If people find value in the section of the regtests that talk to real
> cloud services (but do not run in CI), we can keep them. Still, we have
> examples of JUnit-based tests that can talk to real cloud services too.
>
> However, this is just my opinion. Let's wait a bit for other people to
> express their opinions. We had a good discussion during the sync call, but
> I'm not sure we have consensus yet... hence this thread.
>
> Cheers,
> Dmitri.
>
> On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org> wrote:
>
>> One clarification, the regtests we talk about here is only
>> https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
>> or https://github.com/apache/polaris/tree/main/regtests as well? as the
>> v3.5/regtests doesn't have cloud dependencies but the later one has.
>>
>> Thanks,
>> Yong Zheng
>>
>> On 2026/05/29 01:29:24 Yong Zheng wrote:
>> > Hello Dmitri,
>> >
>> > Thanks for the recap. If I understand correctly, we want to merge test
>> coverage from `/regtests` (docker based) into `/integration`? Looking at
>> them, integration has most of the tests we have in regtests excepts the
>> ones that are real cloud specific such as spark_sql_gcp* and
>> spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud
>> specific ones are not part of CI, in that case, do we still want to covert
>> them? Also, I am not sure if we can convert the pyspark code into JUnit
>> directly.
>> >
>> > Also, regarding those duplicates classes, should we move them to
>> common? For the ones that are very similar but minor diff, should we
>> proceed with adapters for version specific (meaning, we won't be follow
>> what Iceberg is doing with different version of spark).
>> >
>> > Lastly, should we close current PR and handle the two above in a
>> separate PRs first before revisiting Spark4 support?
>> >
>> > Thanks,
>> > Yong Zheng
>> >
>> > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
>> > > Hi All,
>> > >
>> > > This is to recap and follow up on today's Community Sync discussion.
>> This
>> > > matter came up during the review of the Spark 4 PR, but I believe it
>> > > deserves a dedicated discussion.
>> > >
>> > > Observations:
>> > >
>> > > * Tests under the /regtests directory take a considerable amount of CI
>> > > resources. This is mostly due to building custom Docker images.
>> > >
>> > > * When adding support for newer Spark versions, developers naturally
>> tend
>> > > to copy existing test infrastructure, which results in building more
>> Docker
>> > > images.
>> > >
>> > > * The coverage these tests provide probably does not require the heavy
>> > > docker machinery. These tests validate that the Spark Session can
>> interact
>> > > with Polaris over the Iceberg REST Catalog Java client. The same
>> coverage
>> > > can be provided in an isolated JVM running under Gradle more
>> efficiently
>> > > (without a docker env.)
>> > >
>> > > Proposal:
>> > >
>> > > * Gradually convert Spark tests under /regtest to Gradle tasks and
>> JUnit
>> > > tests without Docker.
>> > >
>> > > The exact impl. is to be figured out.
>> > >
>> > > Most of the tests can run under the Junit framework using a local
>> Polaris
>> > > server (same JVM or different JVM depending on use case).
>> > >
>> > > True "integration" tests can still run under Gradle as a task that
>> starts a
>> > > Spark shell in a fresh JVM and executes a small set of SQL commands
>> in it.
>> > > However, these "big" tests can probably be limited in number and
>> complexity
>> > > and as such should not generate excessive CI load. More specifically,
>> these
>> > > tests probably do not need to assert the verbatim output of the SQL
>> > > commands. A basic sanity check should be sufficient. Functional tests
>> can
>> > > be performed under JUnit, I think.
>> > >
>> > > Thoughts?
>> > >
>> > > Thanks,
>> > > Dmitri.
>> > >
>> >
>>
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yong,
Good questions :) As for me I'd refactor all shell-based regtests that use
Docker into Gradle-based tests (preferably JUnit).
If people find value in the section of the regtests that talk to real
cloud services (but do not run in CI), we can keep them. Still, we have
examples of JUnit-based tests that can talk to real cloud services too.
However, this is just my opinion. Let's wait a bit for other people to
express their opinions. We had a good discussion during the sync call, but
I'm not sure we have consensus yet... hence this thread.
Cheers,
Dmitri.
On Thu, May 28, 2026 at 10:00 PM Yong Zheng <yz...@apache.org> wrote:
> One clarification, the regtests we talk about here is only
> https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests
> or https://github.com/apache/polaris/tree/main/regtests as well? as the
> v3.5/regtests doesn't have cloud dependencies but the later one has.
>
> Thanks,
> Yong Zheng
>
> On 2026/05/29 01:29:24 Yong Zheng wrote:
> > Hello Dmitri,
> >
> > Thanks for the recap. If I understand correctly, we want to merge test
> coverage from `/regtests` (docker based) into `/integration`? Looking at
> them, integration has most of the tests we have in regtests excepts the
> ones that are real cloud specific such as spark_sql_gcp* and
> spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud
> specific ones are not part of CI, in that case, do we still want to covert
> them? Also, I am not sure if we can convert the pyspark code into JUnit
> directly.
> >
> > Also, regarding those duplicates classes, should we move them to common?
> For the ones that are very similar but minor diff, should we proceed with
> adapters for version specific (meaning, we won't be follow what Iceberg is
> doing with different version of spark).
> >
> > Lastly, should we close current PR and handle the two above in a
> separate PRs first before revisiting Spark4 support?
> >
> > Thanks,
> > Yong Zheng
> >
> > On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > > Hi All,
> > >
> > > This is to recap and follow up on today's Community Sync discussion.
> This
> > > matter came up during the review of the Spark 4 PR, but I believe it
> > > deserves a dedicated discussion.
> > >
> > > Observations:
> > >
> > > * Tests under the /regtests directory take a considerable amount of CI
> > > resources. This is mostly due to building custom Docker images.
> > >
> > > * When adding support for newer Spark versions, developers naturally
> tend
> > > to copy existing test infrastructure, which results in building more
> Docker
> > > images.
> > >
> > > * The coverage these tests provide probably does not require the heavy
> > > docker machinery. These tests validate that the Spark Session can
> interact
> > > with Polaris over the Iceberg REST Catalog Java client. The same
> coverage
> > > can be provided in an isolated JVM running under Gradle more
> efficiently
> > > (without a docker env.)
> > >
> > > Proposal:
> > >
> > > * Gradually convert Spark tests under /regtest to Gradle tasks and
> JUnit
> > > tests without Docker.
> > >
> > > The exact impl. is to be figured out.
> > >
> > > Most of the tests can run under the Junit framework using a local
> Polaris
> > > server (same JVM or different JVM depending on use case).
> > >
> > > True "integration" tests can still run under Gradle as a task that
> starts a
> > > Spark shell in a fresh JVM and executes a small set of SQL commands in
> it.
> > > However, these "big" tests can probably be limited in number and
> complexity
> > > and as such should not generate excessive CI load. More specifically,
> these
> > > tests probably do not need to assert the verbatim output of the SQL
> > > commands. A basic sanity check should be sufficient. Functional tests
> can
> > > be performed under JUnit, I think.
> > >
> > > Thoughts?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Yong Zheng <yz...@apache.org>.
One clarification, the regtests we talk about here is only https://github.com/apache/polaris/tree/main/plugins/spark/v3.5/regtests or https://github.com/apache/polaris/tree/main/regtests as well? as the v3.5/regtests doesn't have cloud dependencies but the later one has.
Thanks,
Yong Zheng
On 2026/05/29 01:29:24 Yong Zheng wrote:
> Hello Dmitri,
>
> Thanks for the recap. If I understand correctly, we want to merge test coverage from `/regtests` (docker based) into `/integration`? Looking at them, integration has most of the tests we have in regtests excepts the ones that are real cloud specific such as spark_sql_gcp* and spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud specific ones are not part of CI, in that case, do we still want to covert them? Also, I am not sure if we can convert the pyspark code into JUnit directly.
>
> Also, regarding those duplicates classes, should we move them to common? For the ones that are very similar but minor diff, should we proceed with adapters for version specific (meaning, we won't be follow what Iceberg is doing with different version of spark).
>
> Lastly, should we close current PR and handle the two above in a separate PRs first before revisiting Spark4 support?
>
> Thanks,
> Yong Zheng
>
> On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> > Hi All,
> >
> > This is to recap and follow up on today's Community Sync discussion. This
> > matter came up during the review of the Spark 4 PR, but I believe it
> > deserves a dedicated discussion.
> >
> > Observations:
> >
> > * Tests under the /regtests directory take a considerable amount of CI
> > resources. This is mostly due to building custom Docker images.
> >
> > * When adding support for newer Spark versions, developers naturally tend
> > to copy existing test infrastructure, which results in building more Docker
> > images.
> >
> > * The coverage these tests provide probably does not require the heavy
> > docker machinery. These tests validate that the Spark Session can interact
> > with Polaris over the Iceberg REST Catalog Java client. The same coverage
> > can be provided in an isolated JVM running under Gradle more efficiently
> > (without a docker env.)
> >
> > Proposal:
> >
> > * Gradually convert Spark tests under /regtest to Gradle tasks and JUnit
> > tests without Docker.
> >
> > The exact impl. is to be figured out.
> >
> > Most of the tests can run under the Junit framework using a local Polaris
> > server (same JVM or different JVM depending on use case).
> >
> > True "integration" tests can still run under Gradle as a task that starts a
> > Spark shell in a fresh JVM and executes a small set of SQL commands in it.
> > However, these "big" tests can probably be limited in number and complexity
> > and as such should not generate excessive CI load. More specifically, these
> > tests probably do not need to assert the verbatim output of the SQL
> > commands. A basic sanity check should be sufficient. Functional tests can
> > be performed under JUnit, I think.
> >
> > Thoughts?
> >
> > Thanks,
> > Dmitri.
> >
>
Re: [DISCUSS] Future of the "regtests" code
Posted by Yong Zheng <yz...@apache.org>.
Hello Dmitri,
Thanks for the recap. If I understand correctly, we want to merge test coverage from `/regtests` (docker based) into `/integration`? Looking at them, integration has most of the tests we have in regtests excepts the ones that are real cloud specific such as spark_sql_gcp* and spark_sql_azure* as well as pyspark specific (t_pyspark). The cloud specific ones are not part of CI, in that case, do we still want to covert them? Also, I am not sure if we can convert the pyspark code into JUnit directly.
Also, regarding those duplicates classes, should we move them to common? For the ones that are very similar but minor diff, should we proceed with adapters for version specific (meaning, we won't be follow what Iceberg is doing with different version of spark).
Lastly, should we close current PR and handle the two above in a separate PRs first before revisiting Spark4 support?
Thanks,
Yong Zheng
On 2026/05/28 21:07:44 Dmitri Bourlatchkov wrote:
> Hi All,
>
> This is to recap and follow up on today's Community Sync discussion. This
> matter came up during the review of the Spark 4 PR, but I believe it
> deserves a dedicated discussion.
>
> Observations:
>
> * Tests under the /regtests directory take a considerable amount of CI
> resources. This is mostly due to building custom Docker images.
>
> * When adding support for newer Spark versions, developers naturally tend
> to copy existing test infrastructure, which results in building more Docker
> images.
>
> * The coverage these tests provide probably does not require the heavy
> docker machinery. These tests validate that the Spark Session can interact
> with Polaris over the Iceberg REST Catalog Java client. The same coverage
> can be provided in an isolated JVM running under Gradle more efficiently
> (without a docker env.)
>
> Proposal:
>
> * Gradually convert Spark tests under /regtest to Gradle tasks and JUnit
> tests without Docker.
>
> The exact impl. is to be figured out.
>
> Most of the tests can run under the Junit framework using a local Polaris
> server (same JVM or different JVM depending on use case).
>
> True "integration" tests can still run under Gradle as a task that starts a
> Spark shell in a fresh JVM and executes a small set of SQL commands in it.
> However, these "big" tests can probably be limited in number and complexity
> and as such should not generate excessive CI load. More specifically, these
> tests probably do not need to assert the verbatim output of the SQL
> commands. A basic sanity check should be sufficient. Functional tests can
> be performed under JUnit, I think.
>
> Thoughts?
>
> Thanks,
> Dmitri.
>