You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@geode.apache.org by Jacob Barrett <ja...@vmware.com> on 2020/06/30 18:30:20 UTC

Us vs Docker vs Gradle vs JUnit

All,

We are in a bit of a pickle. As you recall from a few years back in an effort to both stabilize and parallelize integration, distributed and other integration/system like test we use Docker. Many of the tests reused the same ports for services which cause them to fail or interact with each other when run in parallel. By using Docker to isolate a test we put a bandage on that issue. The plugin overrides Gradle’s default forked runner by starting the runners in Docker containers and marshaling the execution parameters to those Dockerized runners.

The Docker test plugin is effectively unmaintained. The author seems content on keeping it compatible with Gradle 4. We forked it to work with Gradle 5 and various other issues we have hit over the years. We have shared patches in the past with little luck in having them merged and still its only compatible with Gradle 4.8 at best. I spent some time trying to port it to Gradle 6 but its going to be a larger undertaking given that Gradle 6 is fully Java modules compatible. They added new members throughout to handle modules in addition to class paths.

Long story short because our tests can’t be parallelized without a container system we are stuck. We can’t go to JUnit 5 without updating Docker plugin (potentially minor changes). We can’t go to Gradle 6 without updating the Docker plugin (potentially huge changes). Being stuck is not a good place. I see two paths out of this:

1) We buckle down and fix the tests so they can run in parallel via the normal forking mechanism of Gradle. I know some effort has been expended in this by using our new rules for starting servers. We should need to go further.

2) Fully invest in the Docker plugin. We would need to fork this off as a fully maintain sub-project of Geode. We would need to add to it support for both Gradle 6 and JUnit 5.

My money is on fixing the tests. It is clear, at least from my exhaustive searching, nobody in the Gradle and JUnit communities are isolating their tests with containers. They are creating containers to host service for system level testing, see Testcontainers project. The tests themselves run in the local kernel space (not in container).

We made this push in the C++ and .NET tests, a much smaller set of tests, and it works great. The framework takes care to create clusters that do not interact with each other on the same host. Some things in Geode make this harder than others, like http service not support ephemeral port selection, and gfsh not providing machine readable output about ephemeral port selections. We use port knocking to prevent the OS from assigning the port ephemerally to another process. The framework knocks, opens and then closes, all the ports it needs for the server/locator services and starts them explicitly on those ports. Because of port recycling rules in the OS another ephemeral port request won’t get those ports for some time after they are closed. It's not perfect but it works. Fixing Geode to support ephemeral port selection and a better reporting mechanisms for those port choices would be more ideal. Also, we only start services necessary for the test, like don’t start the http ports if they aren’t going to be used.

I would love some feedback and thoughts on this issue. Does anyone else see a different path forward?

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Jacob Barrett <ja...@vmware.com>.

> On Jul 1, 2020, at 12:16 PM, Udo Kohlmeyer <ud...@vmware.com> wrote:
> 
> To think a little more left field, with our continued investment in K8’s, maybe we can look into that area?
> Run tests in parallel using K8’s?

It’s an interesting idea but even more complicated than the docker solution. Since k8s could schedule the job on any node we have to make sure the build is distributed to all nodes. We could do this with a shared volume or config map. The result collection would be limited to just the test output collected form log data, getting other artifacts from the test would be even more complicated. Probably need yet another shared volume for that, or more than one. Not impossible but pretty complicated workload.

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Udo Kohlmeyer <ud...@vmware.com>.

To think a little more left field, with our continued investment in K8’s, maybe we can look into that area?
Run tests in parallel using K8’s?

But I am also supportive of fixing the tests that we can run them in parallel without the extra container scaffolding.

—Udo
On Jul 1, 2020, 11:38 AM -0700, Jacob Barrett <ja...@vmware.com>, wrote:


On Jul 1, 2020, at 11:14 AM, Kirk Lund <kl...@apache.org> wrote:

I'm not a big fan of forking the Docker plugin and making it a new Geode
submodule. This approach kind of flies in the face of the intentions of OSS
in general. For example, we want folks contributing to Apache Geode rather
than forking Geode to create their own new project while never giving back
to this project.

If the original Docker plugin project refuses to maintain or take on new
contributors then some of us should help lead the creation of a new Docker
plugin repo in Github that is independent of Apache Geode. This way it
becomes a new living Docker plugin project that many contributors can
become involved in. It also becomes valuable to more projects than just
one. I've seen this happen with a number of OSS projects in which the
original maintainer of a project disappears or leaves and I think this is
generally accepted as the best approach for reviving a dead project that no
new committers can be added to.

I agree, if we want to keep using the plugin it does make sense to open source the fork outside of Geode.

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Jacob Barrett <ja...@vmware.com>.


> On Jul 1, 2020, at 11:14 AM, Kirk Lund <kl...@apache.org> wrote:
> 
> I'm not a big fan of forking the Docker plugin and making it a new Geode
> submodule. This approach kind of flies in the face of the intentions of OSS
> in general. For example, we want folks contributing to Apache Geode rather
> than forking Geode to create their own new project while never giving back
> to this project.
> 
> If the original Docker plugin project refuses to maintain or take on new
> contributors then some of us should help lead the creation of a new Docker
> plugin repo in Github that is independent of Apache Geode. This way it
> becomes a new living Docker plugin project that many contributors can
> become involved in. It also becomes valuable to more projects than just
> one. I've seen this happen with a number of OSS projects in which the
> original maintainer of a project disappears or leaves and I think this is
> generally accepted as the best approach for reviving a dead project that no
> new committers can be added to.

I agree, if we want to keep using the plugin it does make sense to open source the fork outside of Geode.

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Kirk Lund <kl...@apache.org>.

I'm not a big fan of forking the Docker plugin and making it a new Geode
submodule. This approach kind of flies in the face of the intentions of OSS
in general. For example, we want folks contributing to Apache Geode rather
than forking Geode to create their own new project while never giving back
to this project.

If the original Docker plugin project refuses to maintain or take on new
contributors then some of us should help lead the creation of a new Docker
plugin repo in Github that is independent of Apache Geode. This way it
becomes a new living Docker plugin project that many contributors can
become involved in. It also becomes valuable to more projects than just
one. I've seen this happen with a number of OSS projects in which the
original maintainer of a project disappears or leaves and I think this is
generally accepted as the best approach for reviving a dead project that no
new committers can be added to.

On Tue, Jun 30, 2020 at 11:30 AM Jacob Barrett <ja...@vmware.com> wrote:

> All,
>
> We are in a bit of a pickle. As you recall from a few years back in an
> effort to both stabilize and parallelize integration, distributed and other
> integration/system like test we use Docker. Many of the tests reused the
> same ports for services which cause them to fail or interact with each
> other when run in parallel. By using Docker to isolate a test we put a
> bandage on that issue. The plugin overrides Gradle’s default forked runner
> by starting the runners in Docker containers and marshaling the execution
> parameters to those Dockerized runners.
>
> The Docker test plugin is effectively unmaintained. The author seems
> content on keeping it compatible with Gradle 4. We forked it to work with
> Gradle 5 and various other issues we have hit over the years. We have
> shared patches in the past with little luck in having them merged and still
> its only compatible with Gradle 4.8 at best. I spent some time trying to
> port it to Gradle 6 but its going to be a larger undertaking given that
> Gradle 6 is fully Java modules compatible. They added new members
> throughout to handle modules in addition to class paths.
>
> Long story short because our tests can’t be parallelized without a
> container system we are stuck. We can’t go to JUnit 5 without updating
> Docker plugin (potentially minor changes). We can’t go to Gradle 6 without
> updating the Docker plugin (potentially huge changes). Being stuck is not a
> good place. I see two paths out of this:
>
> 1) We buckle down and fix the tests so they can run in parallel via the
> normal forking mechanism of Gradle. I know some effort has been expended in
> this by using our new rules for starting servers. We should need to go
> further.
>
> 2) Fully invest in the Docker plugin. We would need to fork this off as a
> fully maintain sub-project of Geode. We would need to add to it support for
> both Gradle 6 and JUnit 5.
>
> My money is on fixing the tests. It is clear, at least from my exhaustive
> searching, nobody in the Gradle and JUnit communities are isolating their
> tests with containers. They are creating containers to host service for
> system level testing, see Testcontainers project. The tests themselves run
> in the local kernel space (not in container).
>
> We made this push in the C++ and .NET tests, a much smaller set of tests,
> and it works great. The framework takes care to create clusters that do not
> interact with each other on the same host. Some things in Geode make this
> harder than others, like http service not support ephemeral port selection,
> and gfsh not providing machine readable output about ephemeral port
> selections. We use port knocking to prevent the OS from assigning the port
> ephemerally to another process. The framework knocks, opens and then
> closes, all the ports it needs for the server/locator services and starts
> them explicitly on those ports. Because of port recycling rules in the OS
> another ephemeral port request won’t get those ports for some time after
> they are closed. It's not perfect but it works. Fixing Geode to support
> ephemeral port selection and a better reporting mechanisms for those port
> choices would be more ideal. Also, we only start services necessary for the
> test, like don’t start the http ports if they aren’t going to be used.
>
> I would love some feedback and thoughts on this issue. Does anyone else
> see a different path forward?
>
> -Jake
>
>
>
>
>
>

Re: Us vs Docker vs Gradle vs JUnit

Posted by Dale Emery <de...@vmware.com>.

> On Jun 30, 2020, at 12:28 PM, Jinmei Liao <ji...@vmware.com> wrote:
> 
> I would vote for fixing the tests to use gradle's normal forking. If we are going to invest time and effort, let's invest in an option that can reduce our dependencies

I agree wholeheartedly!

Dale

Re: Us vs Docker vs Gradle vs JUnit

Posted by Jacob Barrett <ja...@vmware.com>.

> On Jun 30, 2020, at 2:10 PM, Anthony Baker <ba...@vmware.com> wrote:
> 
> When evaluating technical alternatives I think it’s helpful to look at data.  Has anyone recently tried to run the entire dunit test suite in parallel w/o docker?  How many tests need to be changed?  IIRC, there would be non-trivial work in product code around statics and system properties as well.

The issues of globals, singletons and system properties is still an issue in the Docker plugin world and its why we recycle the process in the container after each test, `forkEvery = 1`. We would do the same for plain forked runner.

> Maybe pursuing a dual short-term / long-term approach ends up being the most realistic approach.  

Oh I think that is a must. We won’t be able to address all tests with a short term fix to eliminate docker plugin, nor do we want to stick with docker plugin and never upgrade Gradle or JUnit. I suggest we hit on module and test category at a time and burn down the debt that way. First step would be to find a way to keep the docker configuration from being applied to specific cleaned up test tasks. Then apply that change to one task in one module and test.

> @Jake have you tried using the testcontainer project with dunit?  Maybe it’s possible to use GenericContainer with an open RMI port.

I am not sure how you think that will work. The problem testcontainers solves well is the single port service where the client doesn’t care what the port is. So things like databases, web servers, etc are really easy. Geode is hard because we have multiple ports for multiple services. The services have to talk to each other, so you will need to compose some networking between the containers. The client and the server have to agree on ports given the locator resolution, or we invent service gateway here too.

Are you perhaps thinking about the DUnit VMs concept and running those in one or more containers? That might work, there is some trickiness in getting getting the current built bits into the containers. Again, the testcontainers model is around some static image for a resource, like a database. We would be building a container around the currently built binary and tests or mounting them into the container at least, like the Gradle docker plugin does. This would only address clean DUnit tests and not other integration tests that spin up embedded servers, launch gfsh, etc.

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Anthony Baker <ba...@vmware.com>.

When evaluating technical alternatives I think it’s helpful to look at data.  Has anyone recently tried to run the entire dunit test suite in parallel w/o docker?  How many tests need to be changed?  IIRC, there would be non-trivial work in product code around statics and system properties as well.

Maybe pursuing a dual short-term / long-term approach ends up being the most realistic approach.  

@Jake have you tried using the testcontainer project with dunit?  Maybe it’s possible to use GenericContainer with an open RMI port.

Anthony


> On Jun 30, 2020, at 1:20 PM, Donal Evans <do...@vmware.com> wrote:
> 
> +1 for fixing the tests. It'll be a lot of work, but it'll only be a lot of work once, as opposed to taking on maintenance of our own custom Docker plugin, which will be an ongoing effort and not at all immune from getting broken again at some point in the future.
> ________________________________
> From: Jinmei Liao <ji...@vmware.com>
> Sent: Tuesday, June 30, 2020 12:28 PM
> To: dev@geode.apache.org <de...@geode.apache.org>
> Subject: Re: Us vs Docker vs Gradle vs JUnit
> 
> I would vote for fixing the tests to use gradle's normal forking. If we are going to invest time and effort, let's invest in an option that can reduce our dependencies
> ________________________________
> From: Jacob Barrett <ja...@vmware.com>
> Sent: Tuesday, June 30, 2020 11:30 AM
> To: dev@geode.apache.org <de...@geode.apache.org>
> Subject: Us vs Docker vs Gradle vs JUnit
> 
> All,
> 
> We are in a bit of a pickle. As you recall from a few years back in an effort to both stabilize and parallelize integration, distributed and other integration/system like test we use Docker. Many of the tests reused the same ports for services which cause them to fail or interact with each other when run in parallel. By using Docker to isolate a test we put a bandage on that issue. The plugin overrides Gradle’s default forked runner by starting the runners in Docker containers and marshaling the execution parameters to those Dockerized runners.
> 
> The Docker test plugin is effectively unmaintained. The author seems content on keeping it compatible with Gradle 4. We forked it to work with Gradle 5 and various other issues we have hit over the years. We have shared patches in the past with little luck in having them merged and still its only compatible with Gradle 4.8 at best. I spent some time trying to port it to Gradle 6 but its going to be a larger undertaking given that Gradle 6 is fully Java modules compatible. They added new members throughout to handle modules in addition to class paths.
> 
> Long story short because our tests can’t be parallelized without a container system we are stuck. We can’t go to JUnit 5 without updating Docker plugin (potentially minor changes). We can’t go to Gradle 6 without updating the Docker plugin (potentially huge changes). Being stuck is not a good place. I see two paths out of this:
> 
> 1) We buckle down and fix the tests so they can run in parallel via the normal forking mechanism of Gradle. I know some effort has been expended in this by using our new rules for starting servers. We should need to go further.
> 
> 2) Fully invest in the Docker plugin. We would need to fork this off as a fully maintain sub-project of Geode. We would need to add to it support for both Gradle 6 and JUnit 5.
> 
> My money is on fixing the tests. It is clear, at least from my exhaustive searching, nobody in the Gradle and JUnit communities are isolating their tests with containers. They are creating containers to host service for system level testing, see Testcontainers project. The tests themselves run in the local kernel space (not in container).
> 
> We made this push in the C++ and .NET tests, a much smaller set of tests, and it works great. The framework takes care to create clusters that do not interact with each other on the same host. Some things in Geode make this harder than others, like http service not support ephemeral port selection, and gfsh not providing machine readable output about ephemeral port selections. We use port knocking to prevent the OS from assigning the port ephemerally to another process. The framework knocks, opens and then closes, all the ports it needs for the server/locator services and starts them explicitly on those ports. Because of port recycling rules in the OS another ephemeral port request won’t get those ports for some time after they are closed. It's not perfect but it works. Fixing Geode to support ephemeral port selection and a better reporting mechanisms for those port choices would be more ideal. Also, we only start services necessary for the test, like don’t start the http ports if they aren’t going to be used.
> 
> I would love some feedback and thoughts on this issue. Does anyone else see a different path forward?
> 
> -Jake
> 
> 
> 
> 
>

Re: Us vs Docker vs Gradle vs JUnit

Posted by Anilkumar Gingade <ag...@vmware.com>.

It feels like, first, we should choose right resources/tools that is suited for the task in hand and helps in achieving the expected result (Testing - easier to develop, run, monitor and report); and then invest in that once. Even if it means to add new tools/subroutines in the product.

E.g.:
Best suited for above requirement:
Runtime environment - Containers (?)
Testing framework - Junit
Build tools - gradle
Reporting/logging - (?)
Managing/Monitoring - (?)

-Anil.

On 6/30/20, 1:21 PM, "Donal Evans" <do...@vmware.com> wrote:

+1 for fixing the tests. It'll be a lot of work, but it'll only be a lot of work once, as opposed to taking on maintenance of our own custom Docker plugin, which will be an ongoing effort and not at all immune from getting broken again at some point in the future.
________________________________
From: Jinmei Liao <ji...@vmware.com>
Sent: Tuesday, June 30, 2020 12:28 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Us vs Docker vs Gradle vs JUnit

I would vote for fixing the tests to use gradle's normal forking. If we are going to invest time and effort, let's invest in an option that can reduce our dependencies
________________________________
From: Jacob Barrett <ja...@vmware.com>
Sent: Tuesday, June 30, 2020 11:30 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Us vs Docker vs Gradle vs JUnit

All,

2) Fully invest in the Docker plugin. We would need to fork this off as a fully maintain sub-project of Geode. We would need to add to it support for both Gradle 6 and JUnit 5.

I would love some feedback and thoughts on this issue. Does anyone else see a different path forward?

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Donal Evans <do...@vmware.com>.

+1 for fixing the tests. It'll be a lot of work, but it'll only be a lot of work once, as opposed to taking on maintenance of our own custom Docker plugin, which will be an ongoing effort and not at all immune from getting broken again at some point in the future.
________________________________
From: Jinmei Liao <ji...@vmware.com>
Sent: Tuesday, June 30, 2020 12:28 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Us vs Docker vs Gradle vs JUnit

I would vote for fixing the tests to use gradle's normal forking. If we are going to invest time and effort, let's invest in an option that can reduce our dependencies
________________________________
From: Jacob Barrett <ja...@vmware.com>
Sent: Tuesday, June 30, 2020 11:30 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Us vs Docker vs Gradle vs JUnit

All,

We are in a bit of a pickle. As you recall from a few years back in an effort to both stabilize and parallelize integration, distributed and other integration/system like test we use Docker. Many of the tests reused the same ports for services which cause them to fail or interact with each other when run in parallel. By using Docker to isolate a test we put a bandage on that issue. The plugin overrides Gradle’s default forked runner by starting the runners in Docker containers and marshaling the execution parameters to those Dockerized runners.

The Docker test plugin is effectively unmaintained. The author seems content on keeping it compatible with Gradle 4. We forked it to work with Gradle 5 and various other issues we have hit over the years. We have shared patches in the past with little luck in having them merged and still its only compatible with Gradle 4.8 at best. I spent some time trying to port it to Gradle 6 but its going to be a larger undertaking given that Gradle 6 is fully Java modules compatible. They added new members throughout to handle modules in addition to class paths.

Long story short because our tests can’t be parallelized without a container system we are stuck. We can’t go to JUnit 5 without updating Docker plugin (potentially minor changes). We can’t go to Gradle 6 without updating the Docker plugin (potentially huge changes). Being stuck is not a good place. I see two paths out of this:

1) We buckle down and fix the tests so they can run in parallel via the normal forking mechanism of Gradle. I know some effort has been expended in this by using our new rules for starting servers. We should need to go further.

2) Fully invest in the Docker plugin. We would need to fork this off as a fully maintain sub-project of Geode. We would need to add to it support for both Gradle 6 and JUnit 5.

My money is on fixing the tests. It is clear, at least from my exhaustive searching, nobody in the Gradle and JUnit communities are isolating their tests with containers. They are creating containers to host service for system level testing, see Testcontainers project. The tests themselves run in the local kernel space (not in container).

We made this push in the C++ and .NET tests, a much smaller set of tests, and it works great. The framework takes care to create clusters that do not interact with each other on the same host. Some things in Geode make this harder than others, like http service not support ephemeral port selection, and gfsh not providing machine readable output about ephemeral port selections. We use port knocking to prevent the OS from assigning the port ephemerally to another process. The framework knocks, opens and then closes, all the ports it needs for the server/locator services and starts them explicitly on those ports. Because of port recycling rules in the OS another ephemeral port request won’t get those ports for some time after they are closed. It's not perfect but it works. Fixing Geode to support ephemeral port selection and a better reporting mechanisms for those port choices would be more ideal. Also, we only start services necessary for the test, like don’t start the http ports if they aren’t going to be used.

I would love some feedback and thoughts on this issue. Does anyone else see a different path forward?

-Jake

Re: Us vs Docker vs Gradle vs JUnit

Posted by Jinmei Liao <ji...@vmware.com>.

I would vote for fixing the tests to use gradle's normal forking. If we are going to invest time and effort, let's invest in an option that can reduce our dependencies
________________________________
From: Jacob Barrett <ja...@vmware.com>
Sent: Tuesday, June 30, 2020 11:30 AM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Us vs Docker vs Gradle vs JUnit

All,

We are in a bit of a pickle. As you recall from a few years back in an effort to both stabilize and parallelize integration, distributed and other integration/system like test we use Docker. Many of the tests reused the same ports for services which cause them to fail or interact with each other when run in parallel. By using Docker to isolate a test we put a bandage on that issue. The plugin overrides Gradle’s default forked runner by starting the runners in Docker containers and marshaling the execution parameters to those Dockerized runners.

The Docker test plugin is effectively unmaintained. The author seems content on keeping it compatible with Gradle 4. We forked it to work with Gradle 5 and various other issues we have hit over the years. We have shared patches in the past with little luck in having them merged and still its only compatible with Gradle 4.8 at best. I spent some time trying to port it to Gradle 6 but its going to be a larger undertaking given that Gradle 6 is fully Java modules compatible. They added new members throughout to handle modules in addition to class paths.

Long story short because our tests can’t be parallelized without a container system we are stuck. We can’t go to JUnit 5 without updating Docker plugin (potentially minor changes). We can’t go to Gradle 6 without updating the Docker plugin (potentially huge changes). Being stuck is not a good place. I see two paths out of this:

1) We buckle down and fix the tests so they can run in parallel via the normal forking mechanism of Gradle. I know some effort has been expended in this by using our new rules for starting servers. We should need to go further.

2) Fully invest in the Docker plugin. We would need to fork this off as a fully maintain sub-project of Geode. We would need to add to it support for both Gradle 6 and JUnit 5.

My money is on fixing the tests. It is clear, at least from my exhaustive searching, nobody in the Gradle and JUnit communities are isolating their tests with containers. They are creating containers to host service for system level testing, see Testcontainers project. The tests themselves run in the local kernel space (not in container).

We made this push in the C++ and .NET tests, a much smaller set of tests, and it works great. The framework takes care to create clusters that do not interact with each other on the same host. Some things in Geode make this harder than others, like http service not support ephemeral port selection, and gfsh not providing machine readable output about ephemeral port selections. We use port knocking to prevent the OS from assigning the port ephemerally to another process. The framework knocks, opens and then closes, all the ports it needs for the server/locator services and starts them explicitly on those ports. Because of port recycling rules in the OS another ephemeral port request won’t get those ports for some time after they are closed. It's not perfect but it works. Fixing Geode to support ephemeral port selection and a better reporting mechanisms for those port choices would be more ideal. Also, we only start services necessary for the test, like don’t start the http ports if they aren’t going to be used.

I would love some feedback and thoughts on this issue. Does anyone else see a different path forward?

-Jake